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1.0  INTRODUCTION 

This  study  was  performed  for  the  Defense  Communications  Engineering 
Center  under  Contract  Number  DCA100-77-C-0055.  It  was  a  continuation  of  a 
previous  study  effort  also  conducted  at  Harris  under  Contract  Number 
DCA100-76-C-0028  producing  a  Final  Report^  dated  March  1977. 

The  object  of  these  studies  was  to  select  a  preferred  timing  subsystem 
for  the  future  integrated  switched  digital  Defense  Communications  System  (DCS)  and 
to  quantitatively  justify  the  selected  subsystem.  The  techniques  considered  in 
the  first  study  included: 

t  Master  Slave 

•  Mutual  Synchronization 

•  Time  Reference  Distribution 

•  Independent  Clocks 

•  Pulse  Stuffing 

Pulse  Stuffing  can  be  quickly  discarded  in  a  switched  digital  network 
as  not  being  cost  effective.  The  ability  to  perform  channel  switching  requires 
that  pulse  stuffing,  destuffing  and  rate  jitter  filtering  be  performed  on  a 
channel  by  channel  basis  at  every  multiplexer,  demultiplexer  and  every  time 
division  switch.  This  will  make  this  technique  the  most  expensive  system  by  a 
large  margin.  It  will  not  be  considered  further  in  this  study. 

The  Independent  Clock  technique  does  not  require  very  much  analysis 

except  to  understand  the  free  running  characteristics  of  atomic  clocks.  This  has 
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been  looked  into  in  some  detail  .  This  technique  involves  use  of  a  free 
running  very  high  stability  atomic  clock  at  all  switching  centers  (nodes). 
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Even  when  Cesium  atomic  standards  are  used  there  will  be  some  small  frequency 
difference  between  nodes  causing  the  absolute  phase  difference  between  nodes  to 
accumulate  without  bound.  This  causes  periodic  disruption  in  digital  data  traffic 
and  even  though  the  time  between  disruptions  can  be  limited  to  hours,  this  fact 
primarily  has  been  given  as  the  reason  why  all  digital  communications  systems  both 
commercial  and  military,  except  the  Tri  Tac  system  has  refused  to  consider  its 
use.  In  this  study,  one  of  the  desirable  characteristics  against  which  systems 
were  to  be  evaluated  included  "slip  free  operation  in  its  normal  mode  of 
operation".  This  eliminates  further  consideration  of  the  Independent  Clock 
technique.  While  no  new  analytic  analysis  or  simulations  were  performed  relative 
to  the  Independent  Clock  approach,  since  it  will  be  used  in  the  Tri  Tac 
application,  it  was  treated  as  a  possible  candidate  in  the  design  and  Life  Cycle 
Cost  analysis  performed  as  reported  on  in  Section  4.0  of  this  report. 

An  External  Clock  technique  is  described  in  Section  2.0  of  this 
report.  This  approach  utilizes  timing  derived  from  a  system  like  Loran  C. 

Loran  C  is  extremely  vulnerable  to  distruction  by  enemy  action  making  it  a  poor 
choice  for  a  defense  related  system  in  which  survivability  is  an  important 
consideration. 

The  techniques  remaining  to  be  considered  include  Master  Slave,  Mutual 

pig 

and  Time  Reference  Distribution  (TRD).  In  the  previous  study  Master  Slave 
system  was  defined  as  a  system  in  which  timing  was  distributed  throughout  the 
network  by  derivation  of  the  clock  signal  from  the  communications  received  from  a 
single  path  from  the  master  node  to  any  node  in  the  network.  In  this  follow-on 
study  this  same  approach  will  be  referred  to  as  "the  simplest  form  of  Master  Slave 
synchronization".  In  the  earlier  study,  a  Mutual  technique  was  defined  as  a 
system  in  which  each  node  in  the  network  determined  its  nodal  clock  error  by 
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taking  the  average  of  the  equally  weighted  phase  errors  between  the  local  clock 
and  each  communications  link  into  the  node.  In  this  study  this  technique  will  be 
referred  to  as  the  simplest  Mutual  Sync  technique. 

Time  Reference  Distribution  is  application  of  Master  Slave 
synchronization,  and  by  definition,  TRD  includes  four  specific  refinements  to  the 
simplest  Master  Slave  technique.  The  refinements  or  features  which,  when  added  to 
a  Master  Slave  system  turn  it  into  a  Time  Reference  Distribution  system  are: 

•  Double  Endedness 

•  Independence  of  Clock  Error  Measurement  and  Correction 

•  Phase  Reference  Combining 

•  Self  Organization 

As  defined  in  the  previous  study  neither  Master  Slave  nor  Mutual  Sync 
could  have  any  of  these  features  which  were  solely  reserved  to  be  included  in  TRD. 

The  conclusions  of  the  previous  study  were  questioned  on  the  basis  of 
the  fact  that  a  Master  Slave  sync  approach  to  which  only  the  single  feature  Double 
Endedness  is  added  was  not  considered.  The  Canadian  Dataroute  uses  a  Master  Slave 
technique  with  the  addition  of  the  features  Double  Endedness  and  Self  Organization 
but  not  using  Independence  of  Clock  Error  Measurement  and  Correction  or  Phase 
Reference  Combining.  This  specific  technique  was  not  studied  in  the  earlier  work. 

There  have  been  a  few  proponents  of  Mutual  Sync  who  state  that  the 
previous  study  did  not  evaluate  the  Mutual  Sync  approach  in  which  Double 
Endedness,  Unequal  Weighting,  reference  to  a  system  master  and  similar  features 
could  be  added  to  improve  system  performance  and  give  Mutual  Sync  an  opportunity 
with  these  added  features  to  compete  with  TRD. 
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In  this  study  we  have  generally  tried  to  stop  thinking  of  a  Mutual 
system  and  a  Master  Slave  system  and  instead  to  recognize  that  there  are  some  8 
features  which  can  be  used  in  specifying  a  synchronization  system  approach.  These 
features  are  described  in  Section  2.2.  They  can  be  applied  in  various 
combinations  to  form  a  much  larger  number  of  techniques  than  the  three  basic 
techniques  compared  in  the  previous  report.  If  each  feature  could  be  applied 
regardless  of  whether  any  other  feature  was  applied  then  some  256  different 
systems  might  result.  Fortunately,  some  fetures  can  only  be  applied  if  certain 
other  features  are  included.  In  this  study  some  16  feature  combinations 
represented  all  logical  combinations  of  the  8  described  features.  In  selecting 
these  systems,  features  are  added  one  at  a  time  such  that  comparison  of 
performance  of  the  two  systems  (with  and  without  a  given  feature  with  all  else  the 
same)  would  allow  measuring  performance  improvements  offered  by  each  specific 
feature. 

Section  2.0  provides  detailed  discussion  of  the  synchronization 
problem,  the  generic  approaches,  the  features  and  the  desirable  characteristics  of 
a  system.  This  section  is  introductory  and  background  data. 

Section  3.0  deals  with  the  quantitative  evaluation  of  the  value  of  the 
features  in  their  ability  to  provide  each  of  a  dozen  desirable  characteristics. 

Section  4.0  is  devoted  to  the  detailed  paper  design  and  costing  of  the 
networks  evaluated  in  Section  3.0.  The  designs  were  broken  down  to  parts  lists. 
Chassis  and  cabinet  designs  were  considered  along  with  the  environmental 
requirements  placed  on  hardware  which  must  be  colocated  with  the  eventual  DCS 
timing  and  synchronization  subsystem.  The  cost  for  each  system  considered  was 
obtained  from  the  RCA  Price  Computer  model  and  is  broken  into  Initial  development 
and  hardware,  maintenance  costs  over  20  years  estimated  life  plus  software  costs 
for  the  same  16  combinations  of  system  features  evaluated  in  Section  3.0. 
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Section  5.0  Is  devoted  to  an  attempt  to  delineate  as  completely  as 
possible  all  possible  benefits  as  well  as  penalties  associated  with  distribution 
of  precise  time  in  the  DCS. 

Section  6.0  presents  the  results  of  a  survey  made  of  existing  digital 
switched  communications  systems.  The  Datran,  Canadian  Detaroute,  Western  Union 
and  Bell  systems  were  interviewed.  Each  of  these  organizations  have  or  have 
planned  an  operating  digital  switched  communication  systems.  The  object  of  the 
survey  was  to  determine: 

1.  What  synchronization  technique  was  chosen  and  why? 

2.  What  experience  has  been  gained? 

3.  What  recommendations  would  be  made  for  the  DCS  based  on  this 
experi ence? 

Section  7.0  discusses  the  conclusions  reached  as  a  result  of  this  study 

effort. 
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2.0  TIMING  SUBSYSTEM  CONCEPTS 

The  future  all  digital  Defense  Communications  System  will  be  made  up  of 
many  switching  centers,  called  nodes,  spread  over  the  world.  At  each  node  many 
low  rate  serial  digital  data  streams  must  be  time  division  multiplexed  into  single 
very  high  rate  (in  excess  of  1  Mb/s)  time  division  digital  data  streams  for 
transmission  over  duplex  data  links  between  nodes.  Time  division  multiplexing  is 
a  simple  matter  when  all  digital  timing  is  derived  from  a  comon  frequency  source 
and  there  is  no  variation  in  the  transmission  time  throughout  the  network.  In 
this  ideal  case  all  network  frequencies  are  harmonically  related  and  the  phase 
relationships  between  all  timing  events  throughout  the  network  is  fixed.  When 
nodes  are  separated  by  large  geographical  distances  two  seemingly  independent,  but 
actually  related,  phenomena  make  it  difficult  to  achieve  proper  timing  at  all 
nodes.  The  first  is  the  instantaneous  absolute  phase  difference  between  the  time 
bases  used  to  time  all  functions  at  different  geographical  locations  (nodes).  The 
second  is  the  fact  that  the  propagation  time  in  transmitting  signals  between  nodes 
is  not  zero  and  additionally  is  not  constant  as  a  function  of  time. 

If  the  time  to  propagate  transmitted  signals  were  always  zero  then  a 
periodic  synchronization  signal  arriving  at  all  nodes  simultaneously  could  keep 
all  clocks  in  a  network  in  exact  frequency  and  time  (phase)  lock.  If  propagation 
time  is  allowed  to  be  greater  than  zero  but  assumed  to  be  constant  as  a  function 
of  time  then  measurements  could  determine  transmission  time  to  a  high  degree  of 
accuracy  and  the  time  base  at  various  locations  (nodes)  could  be  adjusted  to  be 
very  precise  relative  to  any  other  location.  In  the  real  world,  however, 
transmission  time  delay  is  not  zero  and  is  not  constant  and  this  contributes 
heavily  to  the  inability  to  determine  the  exact  absolute  difference  between  the 
time  bases  being  generated  at  different  geographical  locations. 
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If,  by  some  external  means,  it  were  possible  to  maintain  a  zero  phase 
difference  between  the  local  clock  reference  at  every  node  in  the  system,  and 
preferably  a  zero  phase  difference  between  each  clock  and  Coordinated  Universal 
Time  (see  Section  5.1.1),  transmission  time  delay  variations  between  nodes  would 
cause  variation  in  the  time  of  arrival  of  data  bits  relative  to  the  absolute  phase 
of  the  hypothetical  "Precise  Time  Clock"  at  each  node.  As  the  transmission  time 
delay  will  not  change  without  bound,  finite,  realizable  elastic  buffers  at  the 
termination  of  each  transmission  link  can  compensate  for  this  variation.  It  is 
convenient  to  consider  a  transmission  path  as  a  delay  line  in  which,  if  the 
duration  of  a  data  bit  is  small  compared  to  the  total  delay  time,  at  any  instant 
in  time  many  bits  exist  in  "in-transit"  storage.  The  transmission  path  appears  to 
the  system  to  be  an  elastic  buffer  but  the  number  of  bits  in  storage  changes  with 
time  as  a  function  of  uncontrollable  physical  phenomena.  Elastic  buffers  at  the 
termination  of  the  links  can  provide  compensating  varying  delay  such  that  the 
apparent  delay  between  input  to  the  transmission  link  and  output  of  the  elastic 
buffer  appears  to  be  a  constant. 

The  accuracy  with  which  two  clocks  can  be  maintained  in  phase  (or  time) 
synchronization  is  a  function  of  the  accuracy  and  frequency  with  which  a  reference 
signal  can  be  transported  between  them.  Since  transmission  time  of  a  radio  signal 
encounters  time  varing  delay  it  imposes  restrictions  on  the  accuracy  of  phase 
synchronization  of  two  clocks  which  is  a  function  of  the  ability  to  measure  or 
compensate  for  delay.  Portable  primary  standards  (cesium  atomic  clocks)  are 
sometimes  physically  carried  between  locations;  however,  the  frequency  of  such 
trips  cannot  be  very  often  and  the  clock  error  during  transportation  cannot  be 
absolutely  zero. 
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The  primary  object  of  a  synchronization  subsystem  in  a  time  division 
multiplex  transmission  system  is  to  provide  timing  signals  throughout  the  network 
such  that  comnunications  can  continue  uninterrupted  for  an  acceptable  length  of 
time.  Interruption  occurs  when  input  data  fails  to  arrive  in  time  to  be  placed  in 
an  outgoing  time  slot  reserved  for  that  data  (buffer  underflow  resulting  from  a 
low  input  data  rate),  or  when  a  time  slot  is  not  available  to  remove  data  from  a 
buffer  before  more  data  arrives  than  the  buffer  can  store  (buffer  overflow 
resulting  from  a  high  input  data  rate).  If  this  is  the  only  definition  of 
requirements  then  there  is  no  need  for  reference  to  precise  time  and  in  fact  the 
frequency  and  absolute  phase,  relative  to  a  primary  standard,  can  be  allowed  to 
change  significantly  as  long  as  all  the  absolute  phase  differences  between  all 
communicating  nodes  are  within  limits  and  elastic  buffers  are  provided  at  the 
termination  of  each  link  to  compensate  for  these  limited  phase  differences,  for  a 
minimum  acceptable  time  interval. 

Such  a  simplistic  definition  may  well  be  sufficient  when  a  network  is 
not  expected  to  operate  in  a  highly  stressed  environment  and  expected  system 
failures  may  be  very  rare.  However  in  a  defense  related  communications  system 
stresses  can  be  expected  to  be  very  great  in  times  of  conflict  with  the 
destruction  of  portions  of  the  network  a  prime  objective  of  enemy  action.  The 
object  of  this  study  was  to  consider  all  refinements  to  methods  of  system 
synchronization  which  may  contribute  to  the  maintenance  of  uninterrupted 
comnunications  taking  into  account  many  expected  natural  as  well  as  enemy 
initiated  stresses  which  may  be  encountered. 

2.1  Generic  Synchronization  Approaches 

There  are  a  number  of  basic  approaches  to  system  synchronization  which 
have  been  used  in  the  past  or  have  been  proposed  and  studied  to  various  degrees. 
These  "Generic"  approaches  carry  many  names  which  will  be  familiar  to  many 
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readers.  In  this  section  these  systems  are  described  showing  that  each  system  may 
be  implemented  in  a  very  basic  way  or  that  various  refinements  can  be  added  to 
enhance  performance. 

For  additional  description  see  References  127,  128,  143,  189  and  other, 
references  referred  to  in  those  papers. 

2.1.1  Independent  Clock  Synchronization 

The  idea  of  continuous  uninterrupted  communications  for  an  acceptable 
length  of  time  is  emphasized  because  of  this  mode  of  synchronization  which  has 
been  chosen  for  the  TRI-TAC  communication  network.  Each  major  node  contains  an 
independent  high  stability  Cesium  Beam  Atomic  Clock  which  runs  totally  independent 
of  all  other  clocks  in  the  network.  The  clocks  at  different  nodes  will  run  at 
slightly  different  frequencies  and  thus  the  absolute  phase  difference  between 
nodes  are  expected  to  accumulate  without  bound.  Elastic  buffers  at  the 
termination  of  every  transmission  link  provide  a  necessary  capacity  to  compensate 
for  the  maximum  predictable  phase  error  which  can  accumulate  between  any  two  nodes 
over  some  acceptable  interval  between  interruptions  in  the  communications.  Use  of 
atomic  clocks  limits  buffers  to  practical  sizes  to  provide  a  mean-time-be tween- 
interruptions  in  excess  of  24  hours  which  is  considered  acceptable  in  the  TRI-TAC 
application. 

Because  of  the  cost  of  atomic  clocks,  they  will  only  be  used  at  major 
nodes  and  Master-Slave  sync  (discussed  in  Paragraph  2.1.3)  will  be  used  at  lower 
level  nodes  and  data  terminals. 

All  other  methods  of  controlling  timing  to  be  addressed  in  this  study 
involve  the  controlling  or  disciplining  of  nodal  clocks  to  run  at  least  at  the 
same  average  frequency  and  thus  limit  the  accumulation  of  absolute  phase 
difference  between  any  two  nodal  clocks  such  that  practical  size  elastic  buffers 
can  provide  uninterrupted  communications,  under  normal  operating  conditions,  for 
an  unlimited  period  of  time. 
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External  Master  Clock  Synchronization 

This  method  of  system  synchronization  is  not  addressed  in  this  study. 
However  it  is  mentioned  because  it  has  been  chosen  as  the  interim  method  of 
synchronizing  those  portions  of  the  DCS  requiring  synchronization  during  the 
transition  to  an  all  digital  network^.  It  is  characterized  by  the  fact  that 
all  timing  and  synchronization  information  used  by  each  node  is  obtained  from  an 
external  system  (unrelated  to  the  communications  system)  which  happens  to  be 
distributing  accurate  timing  information. 

In  the  interim  DCS  sync  approach,  each  node  requiring  synchronization 
will  have  an  AN/GSQ-183  system  installed.  This  equipment  has  receivers  for 
LORAN-C  signal  inputs  and  provides  output  frequencies  which  are  phase  related  to 
the  LORAN-C  transmissions.  Since  all  LORAN-C  chains  generate  signals  which  are 
traceable  to  UTC(USNO)  there  is  an  ultimate  reference  to  a  common  master  clock  and 
thus  phase  errors  between  nodes  will  be  bounded  and  predictable^’^. 

The  greatest  weakness  of  LORAN-C  is  that  the  large  antenna  systems  of 
the  very  few  stations  in  any  particular  area  are  subject  to  destruction  in  a 
wartime  environment,  and  even  if  not  destroyed  the  pulse  transmissions  at  100  kHz 
are  subject  to  jamming  even  from  off  frequency  jammers  located  hundreds  of  miles 
away.  This  makes  the  External  Master  Clock  synchronization  approach,  using 
LORAN-C,  extremely  undesirable  in  a  Defense  related  communications  network. 

Note  that  both  the  independent  clock  and  the  external  master  clock 
synchronization  approaches  make  absolutely  no  use  of  timing  information  which  is 
inherently  contained  in  the  time  division  digital  data  traveling  over  all  paths 
between  nodes.  Actually  in  any  digital  synchronization  system  including 
independent  clock  and  external  master  clock  it  is  necessary  to  detect  the 
transitions  in  data  bits  as  received  over  each  and  every  system  data  link  and  use 
this  timing  to  control  a  local  link  clock  to  allow  regeneration  of  the  data  and 
control  readin  to  the  elastic  buffers.  Also  Time  Division  Multiplex  data  always 
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contains  framing  synchronization  patterns  to  identify  blocks  of  data.  Thus  in  the 


Independent  Clock  and  External  Master  Clock  approaches  of  system  synchronizat ion 
this  inherent  information  concerning  the  frequency  and  event  timing  at  the  other 
end  of  every  data  transmission  link  is  being  ignored,  at  least  as  it  might  relate 
to  determining  a  local  nodal  time  base.  The  remaining  system  synchronization 
modes  to  be  discussed  below  all  make  use  of  this  inherent  availability  of  timing 
information  in  the  time  division  multiplex  data  transmission  to  help  disseminate 
timing  and  synchronization  information  throughout  the  network. 

2.1.3  Master-Slave  Synchronization 

By  far,  the  majority  of  all  time  division  multiplex  transmission 
systems  are  synchronized  using  the  Master-Slave  synchronizati on  approach  (see 
Section  6.0).  In  the  simplest  form  of  Master-Slave  sync,  one  node  is  designated 
as  the  master  node.  All  nodes  which  are  in  direct  communications  with  the  master 
node  derive  their  timing  (local  nodal  clock  frequency)  from  the  data  input  over 
the  direct  link  from  the  master.  Nodes  more  remote  from  the  master  derive  their 
local  nodal  timing  from  one  input  data  link  received  from  a  node  which  is  closer 
to  the  master  node.  There  is  only  one  path  over  which  timing  can  be  received  at 
any  node.  It  is  either  the  shortest  path  to  the  master  node  in  terms  of 
transmission  time  delay  or  the  path  with  the  highest  quality  signal,  which  are 
usually,  but  by  no  means  necessarily,  synonymous.  Master  Slave  synchronization  is 
defined  as  a  system  using  directed  control  in  which  timing  information  is 
distributed  in  only  one  direction  over  any  duplex  link.  The  direction  of  control 
is  away  from  a  single  designated  master  source  or  frequency  reference.  If 
transmission  time  delay  is  not  taken  into  account  the  absolute  time  (or  phase) 
offset  relative  to  the  master  may  well  accumulate  as  the  distance  from  the  master 
increases;  however,  the  long  term  average  frequency  of  all  nodes  is  the  same.  Two 
nodes  which  receive  timing  over  different  paths  from  the  master  can  still  directly 
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communicate  with  bounded  absolute  phase  differences  between  their  clocks.  In  its 
simplest  form  master  clock  failure  or  the  failure  of  a  link  over  which  timing  is 
being  derived  will  cause  total  or  partial  system  failure;  however  automatic 
methods  of  system  monitoring  and  switching  of  control  configuration  have  been 
developedl73,174,215  to  allow  a  master-slave  system  to  perform 
self-reorganization  and  automatic  slaving  to  an  alternate  common  master  node 
and/or  provide  for  dissemination  of  timing  over  alternate  routes. 

Note  that  it  is  not  necessary  that  the  master  node  clock  be  referenced 
to  UTC  or  that  it  even  be  a  highly  stable  frequency  source.  As  long  as  tracking 
time  constants  allow  all  nodes  to  track  the  average  frequency  and  phase  variations 
of  the  master  uninterrupted  digital  communications  can  continue  throughout  the 
network.  However,  when  only  one,  or  a  few  clocks  in  a  network  are  to  be  used  as 
master  clocks,  the  cost  of  very  good  clocks  at  one  or  a  few  locations  almost 
certainly  will  offset  the  cost  of  designing  all  nodes  to  be  able  to  track  a  poor 
quality  master  clock. 

The  master  node  in  a  Master-Slave  synchronization  system  can  itself  be 
slaved,  either  directly  or  indirectly  such  as  through  LORAN-C,  to  UTC(USNO)  and 
result  in  the  system  disseminating  precise  time  to  some  degree  of  accuracy  which 
will  be  heavily  dependent  on  the  transmission  time  delays  in  the  system,  and  the 
sophistication  of  the  hardware  design. 

A  Master-Slave  approach  will  work  making  use  of  only  the  bit  sync 
information  inherent  in  the  data  bit  transitions  and  the  occurrence  of  periodic 
framing  synchronization  patterns.  However  if  only  a  few  hundred  bits  per  second 
of  the  information  capacity  of  the  transmission  links  are  set  aside  to  carry  some 
data  relating  to  the  process  of  synchronization,  a  method  of  timing  control  over  a 
duplex  link  known  as  Double  Endedness  is  possible  which  cancels  out  the  effects  of 
common  mode  transmission  time  delay  of  links  and  makes  the  absolute  phase 
difference  between  clocks  at  the  two  ends  of  the  link  much  smaller.  Double 
Endedness  is  discussed  in  Paragraph  2.2.2. 
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2.1.4  Mutual  Synchronization 

Mutual  Synchronization,  in  its  simplest  form,  involves  controlling  the 
local  nodal  clock  to  be  the  average  of  the  frequency  of  ALL  incoming  data  streams 
to  the  node.  At  each  node  the  absolute  phase  of  the  nodal  clock  is  measured 
relative  to  the  absolute  phase  of  each  link  input  to  that  node  (after  the  input 
has  suffered  the  transmission  time  delay  produced  by  the  link)  and  the  algebraic 
sum  of  these  phase  differences  is  used  to  drive  the  nodal  clock  in  a  direction 
which  minimizes  the  NET  phase  difference.  All  nodal  clocks  contribute  directly  to 
the  control  of  all  directly  connected  nodes,  and  indirectly  to  control  of  all 
other  nodes  in  the  network.  Every  pair  of  nodes  between  which  duplex 
communications  exist  contributes  to  each  other's  control  giving  rise  to  numerous 
feedback  paths  in  a  system  of  many  highly  interconnected  nodes.  Mutual  control  is 
sometimes  referred  to  as  the  opposite  of  directed  control.  Mutual  synchronization 
can  be  defined  as  a  system  which  has  mutual  control,  or  as  one  without  directed 
control.  Such  a  system  has  been  investigated  in  numerous  studies 
188,190,114,40,42,43,45,46  laboratory  models  and  in  at  least  one  case  in  a  small 
operational  digital  communications  systeml96.  it  has  been  shown  that  such  a 
system  can  be  made  stable  and  the  long  term  average  system  frequency  is  a  function 
of  a  weighted  average  of  the  free  running  frequencies  of  all  nodal  clocks  in  the 
network,  plus  a  term  which  is  a  function  of  the  total  transmission  time  delay  in 
all  duplex  links.  Thus,  for  example,  if  a  mutual  sync  system  contains  a  duplex 
link  through  a  satellite  repeater,  the  average  system  frequency  may  vary  rather 
significantly  over  the  daily  vertical  drift  of  the  satellite  which  causes 
significant  changes  in  the  number  of  bits  which  are  in-transit  in  these 
transmission  links. 

An  early  misconception  about  use  of  mutual  sync  was  that  if  a  link  such 
as  a  satellite  link  had  a  large  change  in  transmission  time  delay  (change  in  the 
number  of  data  bits  in  in-transit  storage)  and  this  link  were  imbedded  in  a  large 
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network  of  many  nodes  the  large  changes  in  transmission  link  storage  could  be 
compensated  for  through  the  use  of  many  small  elastic  buffers  throughout  the 
network.  In  other  words  phase  differences  caused  by  transmission  path  length 
change  would  be  equally  distributed  throughout  the  phase  compensating  buffers. 
Further  analysisl88  has  shown  that  this  is  only  true  for  a  trivial  network 
configuration  in  which  the  nodes  are  arranged  in  a  single  circular  pattern.  When 
more  than  one  closed  loop  path  exists  in  the  network  configuration  some  rather 
wild  phase  offsets  can  be  predicted. 

It  was  stated  above  that  in  a  Mutual  Sync  system  the  average  system 
frequency  is  a  weighted  average  of  the  free  running  frequency  of  each  nodal  clock 
in  the  network.  Mutual  Sync  can  be  instrumented  using  Equal  Weighting  (EW).  In 
this  approach  the  phase  differences  between  the  local  clock  and  every  incoming 
conmuni cat ions  link  contribute  equally  to  the  determination  of  the  average  phase 
error  used  to  control  the  local  clock.  However,  even  when  EW  is  used.  Reference 
188  shows  that  the  influence  a  given  node  has  upon  the  average  system  frequency  is 
proportional  to  the  number  of  other  nodes  in  the  network  with  which  that  node  is 
communicating  (i.e.,  a  node  which  is  communicating  with  3  other  nodes  will  have 
three  times  as  much  influence  on  system  average  frequency  as  a  node  which  is 
communicating  with  only  one  other  node).  The  amount  of  influence  each  clock  has 
in  determining  the  final  average  system  frequency  is  directly  a  function  of 
network  topology.  This  might  be  used  to  advantage  by  placing  the  best  quality 
clocks  at  the  most  highly  connected  nodes.  However,  this  influence  changes  when 
nodes  and/or  links  fail  since  the  topology  changes. 

In  the  computer  simulations  reported  in  Section  3.0  some  were  run  with 
equal  weighting  (EW)  of  all  inputs  to  each  node,  while  others  were  run  with 
Unequal  Weighting  (UEW).  When  UEW  was  applied,  three  qualities  of  clocks  were 
assumed  and  when  an  input  to  a  node  was  from  a  node  with  the  highest  grade  of 
clock  it  was  weighted  three  times  more  heavily  than  an  input  from  a  lowest  quality 
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clock  node  in  determining  the  average  phase  error  to  be  applied  to  the  local 
clock.  This  seems  to  imply  that  the  local  node  will  be  influenced  three  times  as 
much  by  the  better  quality  clock  regardless  of  network  topology.  However,  if  the 
highest  quality  clock  is  at  a  node  with  only  one  third  the  number  of  nodes 
cortmuni  eating  with  it  as  are  communicating  with  the  node  in  which  the  lower 
quality  clock  is  located  then  the  above  discussion  indicates  that  each  should  then 
contribute  equally  to  the  average  system  frequency. 

In  the  study  when  different  quality  clocks  were  assigned  to  different 
nodes  in  the  network  the  better  quality  clocks  were  placed  at  highly  connected 
nodes.  This  simply  means  that  their  influence  upon  average  system  frequency  was 
enhanced  both  by  their  greater  weighting  at  the  receiving  node  and  still  further 
enhanced  by  the  fact  that  they  were  at  highly  connected  nodes. 

The  Double  Endedness  Feature  (discussed  in  Paragraph  2.2.2)  can  be 
applied  to  a  Mutual  Sync  System  with  the  result  that  the  average  system  frequency 
will  then  be  completely  independent  of  transmission  time  delay  and  its  variations 
anywhere  in  the  network.  The  average  system  frequency  will  still  be  a  weighted 
average  of  the  free  running  frequencies  of  all  the  nodes  in  the  network.  However, 
for  the  same  system  topology,  the  weighting  factors  will  be  even  greater  for 
highly  connected  nodes  then  when  Double  Endedness  is  not  used.  Again  this  could 
be  used  to  advantage  by  placing  the  highest  quality  clocks  at  the  most  highly 
connected  nodes.  However,  changing  topology  due  to  failure  or  destruction  of 
portions  of  the  system  make  choosing  criteria  for  placement  of  clocks  by  quality 
extremely  difficult. 

The  use  of  Double  Endedness  in  a  mutual  system  does  not  eliminate,  and 
in  fact  Reference  188  shows  that  its  use  will  in  some  instances  increase,  large 
positive  and  negative  phase  offsets  at  some  nodes  in  a  network  when  transmission 
time  delays  change,  depending  upon  network  topology. 
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Still  another  concern  in  using  a  Mutual  Sync  system,  whether  or  not 
Double  Endedness  Is  utilized,  is  that  any  given  topology  will  have  a  Multiplicity 
of  stable  operating  points.  This  phenomenon  is  discussed  in  detail  in  Reference 
188.  A  buffer  underflow  or  overflow  can  result  in  unpredictable  phase  changes 
traveling  to  the  ends  of  the  network  and  returning  until  the  network  finds  a  new 
stable  operating  point  within  the  linear  range  of  every  buffer  within  the 
network.  That  is  true  even  when  the  topology  is  fixed  and  the  overflow  (or 
underflow)  is  caused  by  just  clock  drifts  and  transmission  time  delay  changes. 
Failure  of  nodes  and  or  transmission  links  will  in  many  cases  trigger  the  same 
kind  of  response. 

The  Mutual  Sync  system  has  been  extensively  studied  assuming  that 
unlimited  elastic  buffers  are  available  at  the  ends  of  every  transmission  path. 

The  system  will  find  a  stable  operating  point  and  uninterrupted  communications 
will  continue  until  transmission  time  delay  changes  (which  are  bounded)  and/or 
free  running  clock  frequencies  drift  to  the  extent  that  a  buffer  overflows  or 
underflows.  When  this  occurs  it  will  be  almost  impossible  to  predict  in  a  large 
network  how  many  nodes  of  the  network  will  be  forced  to  overflow  or  underflow  or 
how  long  it  will  take  for  the  network  to  settle  to  a  new  stable  operating 
condition. 

It  has  often  been  stated  that  Mutual  Sync  systems  are  inherently 
self-organizing.  It  is  only  necessary  to  detect  the  failure  of  an  input  to  a  node 
and  eliminate  that  input  from  contributing  to  the  error  averaging  to  control  the 
local  clock  to  automatically  allow  the  network  to  adjust  to  a  new  topology.  This 
statement  is  true  if  and  only  if  the  network  is  not  relying  upon  unequal  weighting 
to  emphasize  the  influence  of  better  clocks,  or  upon  some  system  master  as 
discussed  in  the  next  section.  Such  a  system  will  only  smoothly  settle  to  a  new 
topology  if  all  elastic  buffers  in  the  network  remain  within  bounds  (do  not 
underflow  or  overflow). 
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2. 1.4.1  Mutual  Synchronization  With  Single-Master  Reference  Clock 

This  approach  is  really  a  mixture  of  two  approaches.  One  node  in  the 
network  contains,  or  is  referenced  to,  a  standard  frequency.  The  links  connecting 
that  master  to  all  directly  connected  nodes  operate  under  directed  control  (the 
master  node  cannot  be  influenced  by  its  neighbors).  All  other  links  in  the 
network  operate  under  mutual  control.  Naming  such  a  system  a  mutual  sync  system 
with  a  single  master  reference  is  somewhat  misleading.  It  might  better  be  called 
a  master  slave  system  in  which  timing  distribution,  below  the  level  of  links  which 
are  directly  connected  to  the  master,  is  carried  out  using  mutual  control. 

In  such  a  system,  if  the  average  frequency  of  all  the  nodes,  other  than 
the  master  reference,  differed  from  the  master  reference  then  the  phase  difference 
between  the  master  and  connected  nodes  would  accumulate  unbounded  phase 
differences;  however,  since  nodes  directly  connected  to  the  master  are  driven 
toward  a  zero  net  phase  difference,  this  network  must  obviously  track  the  master. 

This  system  cannot  be  automatically  self-organizing  since,  if  the 
master  clock  fails,  any  other  clock  in  the  network  which  is  to  assume  the  status 
of  master  must  be  told  to  stop  controlling  its  clock  as  a  function  of  input  phase 
differences  and  free  run.  Thus  overhead  data  space  is  required  to  carry  system 
status  information  to  make  such  a  system  self-organizing. 

2.1.5  Time  Reference  Distribution  Synchronization 

173 

Time  Reference  Distribution  Synchronization  is  a  name  applied  to  a 

Master-Slave  Synchronization  approach  to  which  a  number  of  specific  features  have 

been  added  which  improve  the  accuracy  and/or  the  speed  at  which  a  network  can 

bring  all  nodal  clocks  into  phase  (and  thus  also  frequency)  synchronism  with  the 

master  clock  node.  The  five  features  are  described  in  detail  in  Paragraphs  2.2.1 
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through  2.2.5.  Actually  an  earlier  system  called  Time  Reference  Distribution 
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was  defined  which  did  not  contain  the  Phase  Reference  Combining  Feature.  More 
recently  that  feature  was  added  and  to  distinguish  it  from  the  earlier  system  was 
given  the  title  Improved  Time  Reference  Distribution. 

2.2  Timing  Subsystem  Features 

The  following  features  can  be  incorporated  into  design  of  the  timing 
and  synchronization  subsystem  of  a  Time  Division  Multiplex  communications 
network.  Each  feature  enhances  overall  performance  in  some  specific  manner  as  is 
described  in  the  following  subparagraphs.  After  reviewing  the  definition  of  each 
feature,  it  will  become  evident  that  by  definition,  neither  an  independent  clock 
nor  an  External  Master  Clock  synchronization  approach  can  have  any  of  these 
features.  A  Master-Slave  approach  by  definition  has  the  first  feature  and  a 
Mutual  Sync  approach  by  definition  cannot  have  the  first  feature.  The  feature  of 
Double-Endedness  can  be  applied  to  either  Master-Slave  or  Mutual  sync.  It  can  be 
argued  that  some  forms  of  Mutual  sync  inherently  have  the  ability  to  self-organize 
while  this  capability  can  be  added  to  a  Master-Slave  approach  whether  or  not  other 
features  are  included.  Thus  these  features  can  be  applied  to  a  synchronization 
subsystem  in  certain  combinations  to  form  the  Generic  Synchronization  approaches 
discussed  in  Paragraph  2.1  with  certain  logical  mixes  of  the  features  providing  a 
number  of  additional  variations  on  those  described  in  Paragraph  2.1. 

2.2.1  Directed  Control 

Directed  Control  means  that  one  end  of  a  duplex  link  acts  as  a  source 
of  timing  information  and  the  other  end  of  the  link  utilizes  timing  information 
derived  from  the  source  over  that  duplex  link.  The  source  end  of  the  link  is 
either  a  clock  master  or  is  itself  slaved  to  a  clock  master  over  one  or  more 
additional  duplex  links.  It  is  the  fundamental  concept  behind  the  definition  of 
Master-Slave  synchronization  in  which  the  timing  information  established  by  a 
Master  Reference  frequency  source  is  disseminated  only  downward  from  the  master 
node  directly  to  second  level  nodes  and  indirectly  through  higher  level  nodes  to 
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lower  level  nodes.  Directed  control  implies  that  direction  or  designation  of 
routing  of  timing  information  will  involve  the  best  quality  (usually,  though  not 
necessarily  the  shortest)  paths  and  the  avoidance  of  closed  feedback  paths  in 
which  a  change  in  a  clock  can,  after  some  delay  in  traveling  around  some  loop, 
again  affect  the  same  clock.  Directed  control  is  totally  inconsistent  with  Mutual 
Sync  whose  basic  definition  implies  that  timing  information  is  utilized  at  both 
ends  of  every  link  to  contribute  to  the  control  of  the  nodal  clocks. 

2.2.2  Double-Endedness 

Double-Endedness  is  a  technique  in  which  the  phase  error  measurements 
made  at  each  end  of  a  duplex  link  are  encoded  and  inserted  into  overhead  data 
space  in  the  communications  channel  and  sent  to  the  other  end  of  the  duplex  link 
where  it  is  used  to  remove  some  of  the  system  errors  produced  by  transmission  time 
delay  between  the  nodes. 

and  Tg  are  defined  as  the  absolute  instantaneous  phase  of  the 
clocks  at  nodes  A  and  B  respectively.  Note  that  these  are  phases  of  clocks  at 
their  respective  nodes  relative  to  some  hypothetical  perfect  clock.  and  Tg 
are  never  really  known  exactly  with  respect  to  anything  except  themselves,  at 
their  own  nodes.  D^g  and  DgA  are  defined  as  the  transmission  time  delay  from 
Nodes  A  to  B,  and  from  B  to  A  respectively.  At  node  A  a  phase  diffenece 
measurement  (K^)  can  be  made  between  the  instantaneous  phase  of  the  node  A  clock 
and  a  corresponding  phase  of  the  signal  received  from  node  B.  However,  note  that 
this  measurement  is  made  after  the  node  B  clock  has  traversed  the  internodal 
transmission  path  and  has  encountered  the  time  delay  DgA,  Thus  the  actual 
measurement  represents  the  diffence  between  and  (Tg  +  DgA). 


ka  ■  ta 


<TB  *  °BA> 


Similarly,  a  phase  measurement.  Kg,  can  be  made  at  node  B  between  its 
clock  and  the  signal  received  from  node  A  over  the  duplex  path  carrying  the 
coummuni cat  ions  signal  from  node  A  to  node  B. 
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kb  =  Tb  -  (ta  +  dab) 

To  instrument  Double  Endedness  at  node  B  requires  that  the  KA 
measurement  made  at  node  A  be  suitably  encoded  and  inserted  in  the  communications 
link  between  node  A  and  node  B  so  that  it  is  known  at  node  B.  Then  node  B  can 
calculate  the  error  between  its  own  clock  and  the  node  A  clock. 


tb  -  TA  -  (Kb  -  ka)/2  +  (Dab  -  dba)/2 

Three  things  limit  the  accuracy  of  Tg  _  ta.  The  first  is  the 
accuracy  of  each  of  the  KA  an(j  Kg  measurements  which  could  be  made  negligible 
relative  to  other  errors.  The  second  limit  applies  only  if  the  rate  of  change  of 
phase  at  nodes  A  and  B  (the  frequencies)  are  changing.  In  this  case  the 
measurement  sent  from  node  A  to  node  B  (KA)  represents  the  difference  between 
the  node  A  clock  at  a  time  DAg  ear-|jer  and  the  node  B  clock  at  a  time  Dba 
before  that.  This  limits  the  rate  at  which  the  difference  measurement  can  follow 
clock  frequency  changes  at  both  nodes.  However,  in  a  synchronization  subsystem 
time  constants  on  the  control  mechanisms  should  be  very  long  compared  to  the  round 
trip  delay  between  nodes.  Thus,  if  the  errors  (KA  an(j/or  Kb)  are  sent 
frequently  enough  this  error  can  be  made  negligible. 

The  third,  and  most  significant,  error  in  the  clock  difference 

calculation  Tg  _  Ta  will  be  contributed  by  the  differences  in  transmission 
time  delays  between  the  path  from  node  A  to  node  B,  DAB  anc|  from  nocje  b  to  node 

A»  °BA-  Note  that  if  these  delays  were  exactly  equal,  the  phase  error  between 
node  B  and  node  A  would  reduce  to: 

TB  -  TA  -  (KB  -  KA)/2. 

It  is  then  not  necessary  for  the  delays  in  the  duplex  links  to  be  known  if  it  is 
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Consider  in  an  extreme  worst  case  that  node  A  was  in  Chicago  and  node  B 
in  New  York,  and  that  the  link  from  node  A  to  node  B  was  over  a  coaxial  cable. 

D^g  would  be  in  the  range  of  5  milliseconds.  Now  assume  the  link  from  New  York 
to  Chicago  was  through  a  synchronous  satellite.  Its  nominal  delay  would  be  in  the 
range  of  240  milliseconds.  The  difference  in  delay  term  in  Tg  -  TA  would  be 
over  100  milliseconds. 

Fortunately  it  will  be  most  convenient  in  almost  all  cases  to 

instrument  the  two  paths  of  every  duplex  link  using  the  same  transmission  medium 

(i.e.,  both  through  a  satellite,  both  coax  cable,  etc.).  A  recent  program  of 
223 

measurements  of  differential  time  delay  in  various  transmission  media  (where 
both  links  of  a  duplex  link  used  the  same  medium)  indicated  that  differences  were 
in  tens  of  nanoseconds.  Thus  assuming  equal  delays  in  duplex  links  will 
contribute  negligible  error  to  Tg  -  T^  measurements,  at  least  compared  to 
other  expected  system  errors. 

2.2.3  Independence  of  Clock  Error  Measurement  and  Correction 

The  philosophy  behind  this  feature  is  that  the  control  of  the  nodal 
clock  involves  a  filter  time  constant  which  is  measured  in  tens  of  seconds  or 
minutes  or  possibly  even  hours.  As  a  result  when  any  transient  or  transitional 
conditions  occur  there  may  well  be  an  error  control  voltage  input  to  the  clock 
control  loop  which  is  attempting  to  drive  the  local  clock  toward  a  new  operating 
point.  However,  because  of  the  loop  filter  the  clock  is  not  changed 
instantaneously  and  some  "measured  but  uncorrected  error"  is  represented  in  the 
clock  control  input. 
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Assume  Node  A  is  directly  connected  to  the  master  reference  node. 

Using  Double  Endedness  measurements  the  phase  error  between  Node  A  and  the  master 

node  is  known  and  produces  a  measured  but  uncorrected  error  input  to  the  control 

loop  at  Node  A  but  the  clock  at  Node  A  does  not  eliminate  this  error 

instantaneously.  Also  assume  Node  8  is  deriving  its  clock  input  from  Node  A. 

Using  Double  Endedness  measurements  Node  B  knows  its  error  relative  to  Node  A,  but 

at  this  point  Node  B  does  not  know  its  error  relative  to  the  master.  In  a 

standard  Master-Slave  approach  Node  B  is  attempting  to  correct  it  clock  relative 

to  the  signal  received  from  the  Node  A  clock.  If  the  "measured  but  uncorrected 

error"  at  Node  A  (which  is  relative  to  the  master  node)  is  encoded  and  inserted 

into  the  overhead  data  space  in  the  information  capacity  of  each  outgoing  link 

then  Node  B  can  algebraically  add  this  error  to  the  error  obtained  in  the  Double 

Endedness  calculation  and  determine  its  "measured  but  uncorrected  error"  relative 

to  the  master  node.  When  all  nodes  pass  this  kind  of  information  on  to  all 

connected  nodes,  all  nodes  can  calculate  clock  control  inputs  which  represent 

"measured  but  uncorrected  errors"  relative  to  the  system  master  node. 

2.2.4  Phase  Reference  Combining 

In  conventional  Master  Slave  systems,  as  in  the  operational  Canadian 

Dataroute  (described  in  Section  6),  system  clock  is  derived  at  each  node  over  a 

single  path  from  the  master  node.  Rules  can  be  established  such  as  those 

215 

described  by  Darwin  and  Prim  which  allow  the  best  and/or  the  shortest  path  to 
be  automatically  selected  and,  with  nodes  rated  in  terms  of  their  individual  clock 
quality,  provide  for  the  ability  to  reorganize  and  reference  the  best  available 
clock  in  case  of  clock  or  link  failures.  In  a  highly  interconnected  network  there 
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are  a  large  number  of  paths  over  which  clock  information  is  available.  If  the 
information  over  multiple  paths  is  optimumly  combined  it  should  be  possible  to 
establish  a  greater  timing  accuracy  at  all  nodes  in  the  network.  See  Stover227. 

Each  transmission  link  will  introduce  some  measurement  error  and  some 
differential  delay  errors.  For  a  large  number  of  links  of  the  same  type,  these 
errors  can  be  assumed  independent  with  a  mean  value  of  zero,  and  the  error  can  be 
characterized  by  the  standard  deviation  (or  variance)  of  the  expected  error. 
Because  of  these  assumptions  the  errors  statistically  add  as  the  square  root  of 
the  sum  of  the  squares. 

Assume  there  are  two  paths  between  a  node  and  the  master.  Let  the 
measurement  made  over  the  first  path  be  =  y  +  Ej  where  V  is  the  true  value 
and  is  the  error  introduced  by  the  first  path.  Similarly,  the  measurement 
over  the  second  path  is  =  v  +  E2.  It  is  desirable  to  weight  and  combine 
these  measurements  such  as  to  obtain  the  statistically  most  accurate  measurement. 

Mab  *  WiMj  +  W2M2  =  Wi(V  +  Ex )  +  W2(  V  +  E2)  (2.2. 4-1) 

The  weighting  factors  an(j  w2  apply  to  both  the  true  values  and 

the  errors.  The  weighted  true  values  add  linearly,  but  the  weighted  error  values 
(being  assumed  random)  add  as  the  square  root  of  the  sum  of  the  squares 


Mab  *  (W1  +  W2>  V  +  \/VEl2  +  W2%2  (2. 2. 4-2) 
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Since  it  is  desired  that  the  combined  result  Mab  be  the  true  value 
with  a  statistically  minimum  error,  Wj  pius  w?  must  equal  1,  and  the 
expression  under  the  radial  sign  must  be  made  minimum  by  the  selection  of  an(j 

W2.  Substituting  (1-Wi)  for  W2  in  (2. 2. 4-2)  and  finding  the  value  of  Wj 
that  minimizes  the  statistical  error  gives  (2. 2. 4-3)  and  (2. 2. 4-4) 

W1  =  E22  '  (E12  +  E22)  (2. 2.4-3) 


(2. 2. 4-4) 


(2. 2. 4. -5) 


Using  this  combined  value  as  one  member  of  a  new  parallel  pair,  computing  a  new 
combined  value,  and  repeating  this  procedure  until  the  total  number  of  parallel 
pairs  are  included,  the  desired  weighing  factor  for  the  measurement  from  path  p  of 
n  parallel  paths  is 


The  resulting  statistical  error  for  the  combined  measurement  based  on  n  parallel 
paths  is 

Ec  ■  1  /Vlf,  (1/E?) 

The  errors  attributed  to  each  link  are  established  during  engineering 
design  and  includes  an  effect  due  to  differences  in  transmission  time  delay  in  the 
two  directions  of  the  duplex  link  (including  hardware  delays)  and  expected  errors 
in  the  equipment  used  to  measure  timing  differences  between  received  signals  and 
the  local  clock. 
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2.2. 5  Self-Organization 

In  any  network  in  which  a  master  clock  is  designated  it  is  necessary  to 
establish  rules  for  determining  the  source  of  timing  information  to  be  used  by 
each  node.  The  only  clock  disciplined  synchronization  approach  which  does  not 
require  this  information  is  a  Mutual  Sync  system  (without  a  master)  and  this  gives 
rise  to  arguments  that  a  Mutual  Sync  system  is  inherently  self-organizing. 

However,  the  Mutual  Sync  approach  is  considered  extremely  fragile  due  to  the 
interaction  of  gains,  delays  and  topology  causing  Bell  System  engineers  to  make 
the  statement  that  its  use  would  require  a  very  high  degree  of  monitoring  to  allow 
practical  system  management . 

The  Datran,  Western  Union  and  Bell  System  use,  or  plan  to  use,  manual 

control  of  a  Master  Slave  organization.  The  Canadian  Dataroute  uses  automatic 

self-organization  by  selecting  the  best  available  clock  over  the  single  best 

215 

available  path.  It  uses  rules  similar  to  those  esablished  by  Darwin  &  Prim 

To  accomplish  self-organization  in  the  time  Reference  Distribution 
173  227 

approach  Stover  •  has  defined  a  set  of  information  to  be  inserted  in 
overhead  data  space  in  all  outgoing  links  from  every  node  in  the  network.  This 
information  is  used  to  implement  a  set  of  rules  at  all  nodes  to  allow  the  system 
to  organize  its  distribution  of  timing  into  tne  best  possible  configuration  based 
on  that  equipment  which  is  operational  in  the  network. 

Any  system,  such  as  mutual  synchronization,  in  which  an  output  of  a 
node  can,  after  being  processed  by  other  nodes,  contribute  to  the  input  to  that 
node  tends  to  have  stability  problems  and  in  general  reprocessing  of  the  same 
information  should  add  nothing  to  accuracy.  One  solution  is  to  only  accept  timing 
information  from  nodes  which  are  closer  to  the  master  node,  or  higher  in  the 
hierarchy.  However,  this  would  eliminate  the  use  of  information  available  from 
adjacent  nodes  which  happen  to  be  at  the  same  level  in  the  hierarchy  even  though 
that  information  may  not  have  been  previously  processed  by  the  node.  To  make 
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maximum  use  of  all  available  paths  at  a  node  and  still  eliminate  timing  feedback 
loops,  two  classes  of  information  are  defined.  Class  1  timing  information  and  its 
associated  estimated  errors  is  that  information  received  only  from  nodes  higher  in 
the  hierarchy  than  the  local  node.  Class  2  timing  information  is  the  Class  1 
information  combined  with  the  information  from  nodes  at  the  same  level  in  the 
hierarchy.  If,  when  deriving  Class  2  timing  information  a  node  uses  Class  2 
information  from  nodes  higher  in  the  hierarchy  but  is  restricted  to  only  Class  1 
information  from  nodes  at  the  same  level,  the  closed  loop  timing  paths  will  be 
avoided. 

There  are  seven  pieces  of  information  sent  out  by  each  node  over  all 

data  links. 

INFO  1.  Rank  of  the  clock  used  as  the  master  time  reference  for  this  node. 

(This  information  is  used  to  assure  that  the  highest  ranking  clock  in 
the  network  is  used  as  master  and  to  establish  the  order  of  succession 
to  master  when  a  master  fails). 

INFO  2.  Number  of  links  between  the  local  node  and  its  master  time  reference 
(This  information  is  used  to  establish  the  hierarchy.) 

INFO  3.  Time  of  the  clock  at  the  remote  end  of  the  link  (including  the  effect 
of  the  time  required  for  the  signal  to  transit  from  the  remote  node  to 
the  local  node)  as  measured  by  the  clock  at  the  local  node.  This  is 
the  information  required  for  implementing  Double-Endedness  as  discussed 
in  Paragraph  2.2.2. 

INFO  4A.  Measured  but  uncorrected  error  in  the  local  clock  based  on  information 
from  those  neighbors  higher  in  the  timing  hierarchy  than  the  local 
node.  The  term  measured  error  as  used  here  includes  errors  obtained  by 
mathematically  combining  other  measurements  and  the  error  in  this 
measured  error  will  be  called  its  inaccuracy.  The  resulting  Class  1 
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error  information  is  used  by  neighbors  not  lower  in  the  timing 
hierarachy  than  the  local  node  and  can  be  used  by  neighbors  at  the  same 
level  to  determine  their  Class  2  errors,  or  by  nodes  at  higher  levels 
to  aid  in  system  monitoring. 

INFO  48.  Same  as  4A,  except  based  on  information  from  those  neighbors  not  lower 
in  the  hierarchy  than  the  local  node  (Class  1  information  from  the  same 
level  and  Class  2  from  higher  levels).  The  resulting  Class  2  error 
information  is  used  by  all  neighbors  lower  in  the  hierarchy  than  the 
local  node  in  determining  their  Class  1  errors. 

INFO  5A.  Estimated  inaccuracy,  stated  as  a  variance  (or  standard  deviation),  of 
the  local  clock  based  on  information  from  all  neighboring  nodes  higher 
in  the  hierarchy  than  the  local  node.  The  resulting  Class  1 
information  is  used  by  all  neighbors  not  lower  in  the  timing  hierarchy 
than  the  local  node,  and  it  can  be  used  by  neighbors  at  the  same  level 
to  determine  weighting  factors  for  combining  INFO  4A  information  from 
their  neighbors  into  a  Class  2  measured  error  as  discussed  in  Paragraph 
2.2.4. 

INFO  5B.  Same  as  5A  except  based  on  information  from  all  neighbors  not  lower  in 
the  timing  hierarchy  than  the  local  node.  This  resulting  Class  2 
information  is  used  by  all  neighbors  lower  in  the  timing  hierarchy  than 
the  local  node  in  determining  weighting  factors  for  combining  INFO  48 
information  into  a  Class  1  measured  error  as  discussed  in  Pa  agraph 
2.2.4. 

Notice  that  the  two  classes  of  information  under  Item  4  and  the  two 
classes  of  information  under  Item  5  are  distinguished  by  the  sources  of 
information  used  to  obtain  them  and  also  by  the  nodes  that  make  use  of  them.  When 
the  rules  for  their  use  are  also  considered,  it  will  be  observed  that  they  prevent 
the  formation  of  closed  feedback  paths. 
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Each  node  applies  the  following  set  of  rules  for  the  use  of  the 
information  received  from  its  neighbors. 

Rule  1  A  node  initially  entering  the  network  will  use  its  own  clock  as 
its  time  reference  until  a  better  reference  can  be  determined.  Its  own  clock 
provides  a  basic  time  reference  to  which  the  node  always  returns  when  it  has  no 
better  reference  available.  Under  these  conditions,  the  local  node  supplies  the 
rank  of  its  own  clock  to  its  neighbors  as  INFO  1. 

Rule  2  The  first  type  of  information  received  from  neighboring  nodes, 
INFO  1,  provides  the  local  node  with  the  rank  of  the  clock  used  as  the  master  time 
reference  by  each  of  its  neighbors.  If  one  or  more  neighbors'  reference  clocks 
outrank  the  local  clock,  the  node  will  select  the  neighbor  referencing  the  highest 
ranking  master  and  use  it  in  determining  its  own  time  reference.  The  rank  of  the 
master  time  reference  used  by  the  selected  neighbor  will  be  supplied  to  all 
neighbors  as  INFO  1,  i.e.,  the  rank  of  the  clock  used  as  master  for  the  local 
node.  Repeated  application  of  this  rule  by  all  nodes  will  result  in  all  nodes 
referencing  the  same  highest  ranking  master  clock. 

Rule  3  If  the  local  node  is  referencing  its  own  clock  there  are  no 
links  between  the  local  node  and  its  master  reference  and  this  information  is 
supplied  to  its  neighbors  as  INFO  2.  The  second  type  of  information,  INFO  2,  as 
received  from  its  neighbors  provides  the  local  node  with  information  about  the 
number  of  links  between  each  neighboring  node  and  that  neighbors's  master  time 
reference.  Unless  the  clock  at  the  local  node  outranks  the  master  reference  of 
all  of  its  neighbors,  the  number  of  links  between  the  local  node  and  the  master  is 
greater  by  one  than  that  of  the  neighbor  selected  by  Rule  2  which  have  the  least 
number  of  links  between  themselves  and  their  master.  This  information  is  supplied 
to  the  neighboring  nodes  as  INFO  2.  Repeated  application  of  this  rule  will  result 
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in  establishing  the  natural  hierarchy.  INFO  2  information  as  trasmitted  to 
neighboring  nodes  and  as  received  from  them  indicates  the  position  in  the 
hierarchy  of  the  local  node  relative  to  each  of  its  neighbors. 

Rule  4  The  third  type  of  information,  INFO  3,  as  received  from 
neighboring  nodes  provides  the  local  node  with  the  time  difference  between  the 
local  clock  and  the  clock  at  each  neighboring  node  (including  the  signal  transit 
time  from  the  local  node  to  the  neighboring  node.)  INFO  3  as  transmitted  to  the 
corresponding  neighboring  node  is  subtracted  from  this  information,  and  the 
difference  is  divided  by  2.  This  provides  a  measurement  of  the  actual  time 
difference  (no  transit  time  included)  between  the  local  clock  and  the  clock  at 
each  neighboring  node. 

Rule  5  Each  neighboring  node  not  higher  in  the  hierarchy  than  the 
local  node  transmits  to  the  local  node,  as  INFO  4A,  the  Class  1  measured  but 
uncorrected  error  to  its  own  clock,  i.e.,  the  error  determined  using  information 
from  that  neighbor's  neighbors  that  are  higher  in  the  hierarchy  than  the 
neighbor.  Similarly,  each  neighbor  higher  in  the  hierarchy  than  the  local  node 
transmits  to  the  local  node,  as  INFO  4B,  the  Class  2  measured  but  uncorrected 
error  in  its  own  clock,  i.e.,  the  error  determined  using  information  from  that 
neighbor's  neighbors  that  are  not  lower  in  the  hierarchy  than  the  neighbor.  As 
received,  this  information  gives  a  measured  but  uncorrected  error  for  each 
neighboring  node.  To  this  is  added  the  difference  between  the  local  clock  and 
each  neighboring  clock  as  determined  by  Rule  4.  The  result  is  a  set  of  error 
measurements  for  the  local  clock  based  on  information  from  each  of  its  neighbors. 
(The  reason  for  using  Class  1  information  from  some  neighbors  and  Class  2 
information  from  others  is  to  avoid  closed  feedback  paths  while  still  making  very 
effective  use  of  the  available  information.) 
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Rule  6  Each  neighboring  node  not  higher  in  the  hierarchy  than  the 
local  node  transmits  to  the  local  node,  as  INFO  5A,  the  estimated  inaccuracy, 
stated  as  a  variance  (or  standard  deviation),  of  its  Class  1  measured  error. 
Similarly,  each  neighboring  node  higher  in  the  hierarchy  than  the  local  node 
transmits  to  the  local  node,  as  INFO  5B,  the  estimated  inaccuracy,  stated  as  a 
variance  (or  standard  deviation),  of  its  Class  2  measured  error.  This 
information,  as  received,  is  the  estimated  inaccuracy  of  the  measured  but 
uncorrected  error  associated  with  each  neighboring  node.  Add  to  each  member  of 
this  set  of  information  (directly  if  stated  as  variances  or  as  the  square  root  of 
the  sum  of  the  square  if  stated  as  standard  deviations)  the  estimated  inaccuracy 
of  the  link  between  each  neighbor  and  the  local  node  as  determined  during 
engineering  design.  The  result  is  a  set  of  inaccuracies  for  the  set  of  measured 
errors  in  the  local  clock  based  on  information  from  each  neighbor. 

The  estimated  inaccuracy  attributed  to  the  link  between  the  local  node 
and  a  neighbor  as  established  during  engineering  design  includes  several 
parameters.  It  includes  an  effect  due  to  the  differences  in  signal  transit  time 
in  the  two  directions  of  the  duplex  link  which  includes  delay  differences  in  the 
transmitters  and  receivers  at  the  two  ends  of  the  link.  It  also  includes 
inaccuracies  in  the  equipment  used  to  measure  timing  differences  between  the 
received  signal  and  the  local  clock. 

Rule  7  From  the  set  of  error  measurements  for  the  local  clock  as 
determined  by  Rule  5,  and  the  associated  inaccuracies  determined  by  Rule  6,  only 
those  for  neighbors  higher  in  the  timing  hierarchy  than  the  local  node  are 
selected.  These  error  measurements  are  combined  according  to  the  equations  of 
Paragraph  2.2.4.,  to  determine  a  Class  1  measurement  of  the  error  in  the  local 
clock,  i.e.,  based  on  neighbors  higher  in  the  hierarchy  than  the  local  node.  This 
is  supplied  as  INFO  4A,  the  measured  error  in  the  local  clock,  to  all  neighbors 
not  lower  in  the  timing  hierarchy  than  the  local  clock. 
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Rule  8  From  the  set  of  inaccuracies  determined  by  rule  6  only  those 
for  neighboring  nodes  higher  in  the  hierarchy  than  the  local  node  are  selected. 
These  are  combined  to  determine  the  inaccuracy  for  the  measured  error  in  the  local 
clock  based  on  information  from  neighbors  higher  in  the  hierarchy  than  the  local 
node.  This  information  is  supplied  as  INFO  5A  to  all  neighbors  not  lower  in  the 
timing  hierarchy  than  the  local  node. 

Rule  9  From  the  set  of  error  measurements  for  the  local  clock  as 
determined  by  Rule  5  and  the  associated  inaccuracies  determined  from  Rule  6,  all 
those  from  neighbors  not  lower  in  the  hierarchy  than  the  local  node  are  selected. 

These  error  measurements  are  combined  to  determine  a  Class  2  measurement  of  the 

error  in  the  local  clock,  i.e.,  based  on  all  those  neighbors  not  lower  in  the 
timing  hierarchy  than  the  local  node.  This  is  supplied  as  INFO  4B,  the  measured 
error  in  the  local  clock,  to  all  neighbors  lower  in  the  timing  hierarchy  than  the 
local  clock. 

Rule  10  From  the  set  of  inaccuracies  determined  by  Rule  6  all  those 
for  neighbors  not  lower  in  the  timing  hierarchy  than  the  local  node  are  selected. 
These  inaccuracies  are  combined  to  determine  the  inaccuracy  for  the  measured  error 
in  the  local  clock  based  on  information  from  all  those  neighbors  not  lower  in  the 
timing  hierarchy  than  the  local  node.  This  inaccuracy  information  is  provided  as 
INFO  5B  to  all  neighbors  lower  in  the  timing  hierarchy  than  the  local  node. 

The  combining  of  information  over  several  different  paths,  in  addition 

to  providing  more  accurate  time  measurements  at  many  nodes  remote  from  the  master, 
reduces  the  need  for  massive  reorganization  of  the  network  following  some  failures 
as  required  when  using  only  the  best  path.  It  also  provides  the  possibility  for 
quantitative  evaluation  of  the  fitness  of  the  timing  subsystem.  Since  each  time 
error  measurement  (the  term  measurement  as  used  here  includes  the  mathematical 
combination  of  measurement  information  from  different  sources)  has  a  corresponding 
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estimated  inaccuracy,  these  time  error  measurements  and  their  corresponding 
inaccuracy  estimates  can  be  used  to  provide  a  quantitative  alarm  system.  This 
leads  to  Rule  11. 

Rule  11  Rule  5  provides  a  set  of  error  measurements  for  the  local 
clock  based  on  information  from  each  neighboring  node.  Rule  6  provides  a 
corresponding  set  of  inaccuracies  for  these  error  measurements.  Rule  9  provides  a 
combined  measurement  for  the  error  in  the  local  clock.  Rule  10  provides  a 
corresponding  inaccuracy  for  the  combined  measurement.  The  combined  measurement 
as  determined  by  Rule  9  is  subtracted  from  each  member  of  the  set  of  error 
measurements  determined  by  Rule  5.  The  resulting  set  gives  the  difference  between 
each  individual  measurement  and  the  combined  measurement.  The  inaccuracy  (stated 
as  a  variance)  determined  by  Rule  10  is  added  to  each  member  of  the  set  of 
inaccuracies  obtained  by  Rule  6  (also  stated  as  a  variance)  and  the  square  root  of 
each  member  of  this  set  is  taken  to  obtain  a  set  of  estimates  of  the  standard 
deviations  of  the  clock  error  measurements  based  on  information  from  each  neighbor 
relative  to  the  combined  clock  error  measurement.  Each  member  of  the  set  of 
differences  between  individual  measurements  and  the  combined  measurement  is 
divided  by  the  estimate  of  the  corresponding  standard  deviation  to  obtain  a 
normalized  set  of  ratios.  The  lowest  level  alarm  could  be  activated  when  the 
ratio  reaches  2.  This  would  not  be  very  significant  because  this  ratio  would  have 
approximately  a  5  percent  probability  of  occurrence  in  a  normally  operating 
system.  A  second  level  alarm,  activated  when  the  ratio  reaches  3,  should  be  quite 
significant  since  its  probability  of  occurrence  in  a  normally  operating  system 
should  be  only  about  0.3  percent.  A  third  level  alarm,  activated  when  the  ratio 
reaches  4,  should  initiate  some  form  of  a  problem  investigation  since  its 
probability  in  a  normally  operating  system  should  be  less  than  0.01  percent. 
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A  fourth  level  alarm,  activated  when  the  ratio  reaches  5,  should  initiate  definite 
corrective  action  since  its  probability  in  a  normally  operating  system  might  be 
expected  to  be  less  than  one  in  a  million. 

2.2.6  Mutual  Control 

Mutual  Control  is  really  the  opposite  or  complement  of  directed 
control.  It  denotes  that  information  for  the  control  of  timing  passes  in  both 
directions  over  a  duplex  path  between  any  two  nodes.  Mutual  Control  is  synonymous 
with  Mutual  Synchronization  which  is  also  referred  to  as  Frequency  Averaging 
Synchronization  and  Organic  Synchronization. 

2.2.7  Dropout  Smoothing 

Dropout  smoothing  is  a  feature  which  is  applied  in  the  mutual  control 
system  to  smooth  out  abrupt  frequency  transients  which  could  occur  as  a  result  of 
reference  link  dropouts.  In  the  mutual  system,  individual  links  may  have 
relatively  large  phase  errors,  even  though  the  composite  phase  error  at  the  node 
in  question  may  be  quite  small,  since  a  weighted  algebraic  sum  of  errors  of  the 
reference  links  with  respect  to  the  local  clock  is  taken.  For  example.  Reference 

Link  1  may  have  a  large  positive  phase  error  +  while  the  sum  of  all  other  errors 
may  be  approximately  to  give  a  composite  phase  error  of  approximately  0. 

If  Reference  1  suddenly  drops  out,  the  composite  phase  error  will  abruptly  change 
from  approximately  0  to  This  will  result  in  a  large  step  change  in  local 

clock  frequency. 

The  above  undesirable  large  frequency  transient  can  be  avoided  if  a 
device  which  remembers  the  phase  error  of  each  reference  link  after  the  dropout  is 
used  on  each  reference.  The  remembered  value  can  then  be  applied  through  a 
decaying  exponential  multiplier  in  lieu  of  zero  value  for  a  period  of  time  to 
allow  the  remaining  reference  errors  to  slowly  readjust. 
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2.2.8  Drop-In  Smoothing 

Drop-in  smoothing  is  a  feature  which  is  applied  in  directed  control 
systems  to  avoid  a  large  frequency  transient  upon  selection  of  a  new  reference 
link  following  loss  of  an  old  reference  link.  This  feature  is  implemented  with 
Type  2  loops  by  adjusting  the  integrator  voltage  to  exactly  cancel  any  difference 
in  phase  error  between  the  new  reference  and  the  old  reference  immediately  prior 
to  the  loss  of  the  old  reference.  If  the  proportional  plus  integral  correction 
terms  with  the  old  reference  were  K0e^  +  yjpj-Q  an(j  the  phase  error  term  of 
the  new  link  with  respect  to  the  local  clock  were  0e2  then  the  integrator 
voltage  is  adjusted  such  that 

K0el  +  ViNTl  =  K0e2  +  VINT2 
or  VINT2  =  ViNTl  +  K(0el  -  #e2). 

In  this  manner  a  smooth  frequency  transition  can  be  made  when  changing 
references.  The  local  clock's  phase  error  relative  to  its  reference  will  change 
slowly  since  the  integrator  has  a  long  time  constant. 

2. 3  Desirable  Characteristics  of  a  Digital  Communications  Timing  Subsystem 

A  large  number  of  desirable  characteristics  can  be  listed  for  a  timing 
subsystem  of  a  large  digital  communications  network.  In  this  study  the  following 
set  of  12  desirable  characteristics  were  specified.  This  study  directed 
consideration  of  synchronization  subsystems  which  incorporated  the  various 
features  described  in  Paragraph  2.2  and  to  evaluate  to  what  degree  those  systems 
would  provide  these  desirable  characteristics. 

2.3.1  Survivability 

The  synchronization  subsystem  is  survivable;  that  is,  the  subsystem 
provides  a  number  of  levels  of  backup  and  fallback  modes  of  operation, 
specifically  including  the  ability  to  function  for  at  least  24  hours  as  in  the 
independent  clock  mode  at  major  nodes  without  any  slips  or  interruptions  in  user 
communications  due  to  the  synchronization  function. 
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Slip  Free  Operation 

The  standard  mode  of  operation  of  the  synchronization  subsystem  is 
slip-free,  i.e.,  it  does  not  require  the  planned  interruption  of  traffic  (loss  of 
bit  count  integrity)  to  reset  buffers. 

2.3.3  Maximum  Frequency  Accuracy  and  Minimum  Phase  Disturbance 

The  synchronization  subsystem  maximizes  frequency  accuracy  and 
minimizes  phase  disturbances  for  a  given  grade  of  clock;  this  maximizes  the  period 
of  time  that  the  system  can  free  run  without  slip  following  failures. 

2.3.4  Clocks  Closer  to  a  Master  Are  Not  Disturbed  by  Perturbations  Further 
Removed  From  the  Master 

The  clocks  at  higher  levels  of  the  hierarchy  cannot  be  disturbed  by 
perturbation  of  either  clocks  or  transmission  facilities  at  lower  levels  of  the 
network,  i.e.,  for  clocks  or  transmission  facilities  further  removed  from  the 
master  than  the  clock  under  consideration. 

2.3.5  Phase  Errors  at  a  Node  Do  Not  Harmfully  Propagate 

An  error  introduced  into  the  clock  at  any  major  or  minor  node  of  the 
network,  except  the  master  node,  does  not  harmfully  propagate  to  the  clocks  at  any 
other  major  node  of  the  network  (i.e.,  does  not  result  in  loss  of  synchronization 
at  the  other  nodes).  Investigate  the  pros  and  cons  of  having  no  error  propagation 
through  the  network. 

2.3.6  Compliance  With  FED-STD-1002 

The  synchronization  method  complies  with  Federal  Standard  1002,  "Time 
and  Frequency  Reference  Information  in  Telecommunication  System"  to  facilitate 
interoperability  between  telecommunications  facilities  and  systems  of  the  Federal 
Government  without  measurably  reducing  survivability  or  flexibility;  or  increasing 
equipment  or  system  complexity  or  cost. 
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Monitorabi 1 ity  and  Detection  of  Impending  Failure 

The  synchronization  subsystem  is  monitorable  at  the  system  level 
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(functional  versus  equipment  monitoring)  to  provide  a  capability  to  detect 
impending  failures  and  to  take  corrective  action  before  slips  can  occur. 

2.3.8  Minimum  Overhead  Devoted  to  Maintenance  of  Timing  Synchronization 
Minimum  overhead  communications  (overhead  for  information  transfer  and 

control)  between  nodes  for  the  purpose  of  maintaining  system  synchronization  is 
desirable.  This  includes  the  magnitude  of  data  exchanged  as  well  as  the 
requirement  for  dedicated  channels  as  opposed  to  time-shared  overhead  capacity. 
Investigate  whether  intermittent  transfer  of  such  information,  if  overhead  is 
used,  is  more  desirable  than  requiring  continuous  information  exchange  by  overhead 
channels. 

2.3.9  Intersystem  Operability  With  System  Using  Independent  Synchronization 

Approaches  c  3  *  “  3 

The  synchronization  subsystem  permits  interoperation  with  other  digital 
communications  system  employing  different  synchronizing  techniques. 

2.3.10  Cost-Effectiveness 

The  synchronization  subsystem  is  cost-effective,  i.e.,  it  maximizes  the 
ratio  of  capability  to  system  cost. 

2.3.11  Se 1 f -Reorg  an i z  at i on 

When  there  is  a  failure  of  the  master  through  which  the  synchronizing 
subsystem  is  coordinated,  the  subsystem  automatically  selects  a  new  master. 

2.3.12  Precise  Time  Availability 

Precise  time  (UTC)  can  be  made  available  to  the  users  of  the  DCS 
without  introducing  any  significant  penalty,  i.e.,  no  measurable  decrease  in 
survivability  or  flexibility;  or  increase  in  equipment  or  system  complexity  and  in 
system  costs. 
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3.0  QUANTITATIVE  ASSESSMENT  OF  THE  VALUES  OF  TIMING  SUBSYSTEM  FEATURES 

This  section  reports  on  the  results  of  Task  1  as  defined  in  the 
contract  Statement  of  Work.  Under  Task  1  the  contractor  shall  evaluate, 
quantitatively  where  possible,  the  capability  of  each  of  the  synchronization 
subsystem  features  discussed  in  Paragraph  2.2  to  provide  each  of  the  desirable 
characteristics  listed  in  Paragraph  2.3.  Results  shall  be  tabulated  in  a  manner 
that  will  lend  itself  to  comparison  of  various  optional  features  relative  to  the 
desirable  characteristics. 

3. 1  Features 

The  various  features  which  can  be  included  in  a  synchronizaton  system 
were  briefly  described  in  Section  2.2.  In  this  section  these  features  will  be 
examined  in  more  detail  indicating  what  benefit  is  to  be  derived  from  their 
inclusion  into  the  system,  and  what  costs  may  be  incurred. 

3.1.1  Directed  Control 

In  directed  control,  (described  in  Section  2.2.1)  timing  information  is 

derived  from  a  single  master  clock.  All  nodes  in  the  network  derive  timing  from 
information  received  over  a  single  path  from  the  master  node;  or  in  the  use  of 
Phase  Reference  Combining  over  multiple  paths.  However,  the  object  of  directed 
control  is  to  deliver  timing  information  to  every  node  over  the  shortest  path  or 
paths  and  thus  when  multiple  paths  are  selected  for  use  in  Phase  Reference 
Combining  the  selection  process  is  specifically  designed  to  eliminate  feedback 
paths.  That  is,  no  node  can  ever  transmit  timing  information  to  itself.  This  is 
an  important  point  as  will  be  seen  later. 

3. 1.1.1  Comparison  of  Type  1  and  Type  2  Loops 

In  a  directed  control  system,  the  nodal  clock  is  derived  by  locking  a 
very  narrowband  phase-locked  loop  to  the  bit  timing  on  one  of  the  incoming 
transmission  links.  As  has  been  discussed  elsewhere^*^  two  fundamentally 
different  types  of  loops  were  considered  for  use  with  the  directed  control 
approach. 
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The  baseline  phase-locked  loop  model  is  shown  in  Figure  3. 1.1.1. 


Figure  3. 1.1.1.  Baseline  Phase-Locked  Loop  Model 

When  the  loop  filter  is  a  simple  low-pass  filter,  i.e.,  F(s)  = 
a/(s.  +  a),  the  loop  is  called  a  Type  1  loop  and  its  closed  loop  transfer  functi 


A  Type  2  loop  is  obtained  by  using  an  integral  plus  proportional  loop 
filter,  i.e.,  F ( s )  =  (s  +  a )/s .  Then  the  closed  loop  transfer  function  becomes 


where 


and 


1 o  J  =  n _ 

s^  +  2  C  w  s  +  uj' 
n  n 

u>n  =  yaKv  as  before, 

c-  v/V^ 


It  is  also  possible  to  employ  a  Type  3  loop  which  contains  an 
additional  integrator  and,  hence,  has  the  ability  to  track  a  frequency  drift  even 
where  there  has  been  a  loss  of  reference.  However,  this  loop  will  not  be  examined 
here  as  it  appears  that  there  is  no  significant  gain  in  performance  over  the  Type 
2  loop  unless  the  loss  of  reference  persists  for  several  days.  In  addition, 
stability  becomes  a  consideration  with  a  Type  3  loop  as  its  root-locus  plot 
disple^ys  regions  of  instability. 

3. 1.1. 2  Selection  of  Loop  Parameters 

Clearly,  the  performance  of  these  loops  will  be  a  function  of  the 
parameters  l  and  o>n.  Before  discussing  the  selection  of  these  parameters, 
however,  it  is  desirable  to  examine  the  performance  requirements  themselves. 

Broadly  stated,  the  objective  to  be  pursued  is  the  filtering  of  input 
frequency  variations  to  provide  a  nodal  clock  which  is  both  stable  and  closely 
approximates  the  network  frequency. 

It  would  be  desirable  to  be  able  to  solve  the  problem  in  a  manner  that 
produces  "optimum"  loop  parameters.  However,  this  would  require  a  suitable 
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definition  of  "optimum,"  a  requirement  which  would  be  difficult,  if  not 
impossible  to  meet.  A  large  number  of  performance  indices  can  be  defined,  all  of 
them  significant,  and  the  selection  of  the  loop  parameters  in  a  manner  that  would 
in  some  sense  optimize  all  of  these  requires  more  constraints  than  we  have  at 
present.  A  "suitable  definition"  would  presumably  supply  these  constraints. 

The  difficulty  of  finding  optimum  loop  parameters  is  greatly 
ameliorated,  however,  by  the  fact  that  the  network  performance  is  actually  rather 
insensitive  to  the  parameters.  That  is,  small  perturbations  in  the  loop 
parameters  around  those  values  that  have  been  chosen  will  produce  little,  if  any, 
change  in  the  overall  performance  of  the  system.  Any  changes  in  the  parameters 
would  quite  likely  produce  "improvements"  in  performance  that  are  in  reality  only 
trade-offs.  In  other  words,  the  improvement  in  one  aspect  of  performance  would  be 
accompanied  by  some  degradation  in  another  and  the  changes  would  be  slight  in 
either  direction. 

A  discussion  of  the  performance  of  the  Type  2  loop  as  a  function  of  the 
loop  parameter  follows.  The  Type  1  loop  can  be  used  in  a  directed  control  system, 
however,  its  primary  utility  is  found  in  the  mutual  control  system  and  hence  its 
discussion  will  be  deferred  until  Paragraph  3.1.2. 

It  will  be  assumed  that  a  phase  detector  that  is  linear  over  several 
cycles  (an  extended-range  phase  detector)  is  used.  This  will  give  improved 
performance  and  should  be  easy  to  implement  with  a  square  wave  clock.  In 
addition,  it  will  be  assumed  that  it  is  sufficient  to  filter  daily  timing  jitter 
by  a  factor  of  10.  This  has  been  determined  by  examination  of  the  physical 
processes  involved216  and  appears  to  be  a  valid  assumption. 

Use  of  a  Type  2  loop  is  highly  advantageous  as  it  provides  for  tracking 
of  a  constant  offset  between  the  reference  frequency  and  the  local  clock's  natural 
frequency  with  zero  phase  error.  The  Type  2  loop  can  be  adjusted  to  provide  good 
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tracking  performance  while  at  the  same  time  taking  advantage  of  the  inherent 
stability  of  the  local  clock.  By  switching  from  narrowband  loop  parameters,  used 
in  the  normal  tracking  mode,  to  wideband  parameters  the  Type  2  loop  can  provide 
for  rapid  acquisition  of  its  reference  at  initial  startup  or  after  restoration  of 
a  reference  that  has  been  lost  for  a  considerable  period  of  time.  However,  the 
acquisition  mode  was  not  used  in  the  simulations  described  in  this  section. 

A  summary  of  the  loop  parameters  for  the  Type  2  loop  to  be  used  in  a 
directed  control  system,  appears  below. 

For  a  more  complete  discussion  of  these  performance  figures,  see 

Appendix  B. 

PERFORMANCE  OF  A  TYPE  2  LOOP 

with  con  =  5.6  x  ID'5  and  £  =  4.0 

Phase  Error  (^s )  Frac.  Freq.  Error 

Input  Peak  Steady-State  Peak  Steady-State 


Phase  Step. 

1  jiS 

1  ms  0 

4.48  x  10"10 

0 

Frac.  Freq. 

Step, 

Af  =  10*9 
*0 

2.12  ms  0 

10-9 

0 

Frac.  Freq. 

Drift 

10'10/day 

0.37  m s  0.37  ms 

-2.45  x  10'12 

0 

i 
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The  Type  2  loop  has  the  characteristic  that  it  produces  amplification 
of  some  jitter  frequencies  when  the  loop  is  running  without  a  reference.  However, 
the  amount  of  amplification  decreases  monotonical ly  with  £  and  at  £  =  4,  for 
example,  the  peak  jitter  amplification  is  only  1.013.  £  cannot  be  increased 

without  bound,  however,  because  as  £  is  increased,  the  loop  becomes  more 
responsive  to  perturbations  in  the  reference.  This  is  clearly  counterproductive 
to  the  purpose  of  the  loop  which  is  to  provide  a  stable  nodal  clock.  With  this  in 
mind,  it  was  decided  to  use  £  =  4  for  the  tracking  mode.  Then  to  filter  daily 

C 

jitter  by  a  factor  of  10,  it  is  necessary  to  choose  =  5.6  x  10  rad/s. 

As  was  indicated  earlier,  vhile  these  parameters  are  not,  in  any  strict 
mathematical  sense,  optimum,  they  do  provide  quite  acceptable  performance. 
Moreover,  minor  adjustments  of  these  parameters  does  not  yield  any  significant 
change  in  performance. 

The  above  parameters  were  selected  to  achieve  the  desired  performance 
with  quartz  clocks.  Due  to  the  higher  level  of  performance  inherent  in  cesium 
clocks,  different  parameters  were  chosen  for  them.  The  parameters  implemented  in 
the  simulator  were 

£  =  2.0  and  <<> n  =  1.12  x  10"'’  rad/s. 

In  actual  practice,  one  would  employ  even  longer  time  constants  for  use 
with  cesium  clocks.  These  longer  constants  were  not  used  in  the  simulations 
because  of  the  computer  run  time  limitations. 

3.1.2  Mutual  Control 

In  this  section  we  address  the  implementation  of  a  mutual  control 
approach  for  network  synchronization.  Many  of  the  trade-offs  are  similar  to  those 
discussed  in  the  previous  section  with  respect  to  the  directed  control  approach. 
These  include  the  decisions  one  must  make  regarding  loop  type,  loop  parameters, 
phase  detector  type,  clock  stabilities,  and  buffer  sizes.  Additional 
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considerations  encountered  only  with  mutual  control  include  the  individual  phase 
detector  weightings,  the  application  of  dropout  smoothing  and  system  stability. 

Briefly  stated,  mutual  control  is  an  approach  to  network  timing  in 
which  each  node  receives  its  timing  information  from  more  than  one  node  and  two 
nodes  may  even  exchange  timing  information. 

Of  particular  significance  is  the  fact  that  due  to  the  two-way  transfer 
of  timing  information,  there  is  feedback  in  the  timing  subsystem.  This 
immediately  raises  the  question  of  system  stability  which  is  discussed  below. 

3. 1.2.1  System  Stability 

Constraints  on  system  parameters  required  to  produce  a  stable  mutual 
control  system  have  taken  two  forms.  First,  one  must  ensure  that  the  relationship 
between  path  delays  and  loop  gain  is  proper.  This  results  in  upper  bounds  on 
gain-delay  products  (loop  gains  and  link  delays)  that  have  been  derived  by  many 
investigators. 108>  40,  116,  45,  46.  These  stability  bounds  are  dependent  on  the 
specific  network  configuration.  However,  the  maximum  value  of  gain  delay  product 
that  can  typically  be  tolerated  is  on  the  order  of  1.  Since  very  narrowband  loops 
(implying  very  small  loop  gain)  will  be  used,  link  delays  of  many  seconds  would  be 
required  before  there  would  be  a  stability  problem.  Thus,  in  a  real  network  one 
would  not  have  to  be  concerned  about  the  gain-delay  products  being  large  enough  to 
cause  network  instability. 

The  other  constraint  needed  to  ensure  stability  is  on  the  loop  filter 
parameters.  Stability  analyses  for  single-ended  mutual  synchronization  systems 
have  shown  that  in  order  for  a  mutual  synchronization  system  to  be  stable,  a 
closed  loop  phase  response  that  satisfies 
flb(j«)  i  i 

-  =  H(o>)  <1,  a)  >  0.  (3. 1.2-1) 

®i(jw) 

is  required. 
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It  can  be  shown  that  Type  2  loops  with  filters  of  the  form  F  (s)  =  (s  +  a)  /  s 
never  satisfy  this  condition  (we  note  that  jH(a)n)|  >  Th^s  means  that 

phase  jitter  at  the  natural  frequency  wn  win  be  amplified  each  time  it  passes 
through  the  timing  regeneration  circuitry  at  a  node.  Since  this  is  a  closed-loop 
system  of  many  interconnected  PLL's,  one  can  see  that  the  amplification  of  phase 
jitter  is  the  source  of  the  instability.  Thus,  the  designer  is  restricted  to  Type 
1  loops. 

One  could  examine  several  Type  1  loops  for  use  in  this  system.  If  no 
loop  filter  is  used,  a  first-order  loop  results.  It  is  obvious  from  the  transfer 
function  that  such  a  loop  satisfies  Equation  (3. 1.2-1).  If  the  loop  has  a  single 
low-pass  filter  of  the  form  F(s)  =  a/  (s  +  a),  then  one  can  show  that  the 
closed-loop  transfer  function 

»o(s)  . 

®i^  s2  +  2  £  w  s  +  to  2 
n  n 

where  <*>n  =  an(j  f  =  ya/4Kv 

satisfies  Equation  (3. 1.2-1)  provided  that  C  is  greater  than  \/yfT. 

Another  loop  filter  that  yields  a  Type  1  loop  is  F(s)  =  |||  *  * 

However,  it  has  been  shown218  that  this  loop  filter  provides  stable  network 
operation  only  for  a  very  narrow  range  of  loop  parameters.  This  makes  its  use  in 
the  DCS  impractical . 

Thus,  due  to  stability  considerations,  we  will  consider  only  loop 
filters  of  the  form  F(s)  =  *  a  for  use  in  mutual  control  systems.  Design 

criteria  for  this  type  of  loop  are  discussed  in  the  following  subsection. 

3. 1.2. 2  Type  1  Loop 

As  in  the  discussion  of  the  directed  control  system,  we  will  make  the 
assumptions  of  an  extended  range  phase  detector  and  the  sufficiency  of  filtering 
daily  jitter  by  a  factor  of  10. 
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Before  selecting  the  loop  parameters  which  were  actually  used  in  the 
simulation,  four  fundamental  aspects  of  performance  were  given  consideration. 

(See  Appendix  B  for  more  detailed  treatment.) 
i.  Tracking  Performance 

To  track  a  constant  frequency  offset,  the  Type  1  loop  requires 

a  non-zero  steady-state  phase  error  which  it  is  clearly  desirable  to 

minimize.  This  phase  error  decreases  with  decreasing  £  and/or 

increasing  a> 

3  n- 

To  hold  a  local  clock  having  a  linear  natural  frequency  drift, 
P  ,  to  a  fixed  reference,  the  Type  1  loop  requires  a  steady-state 
frequency  error  and  a  linearly  increasing  phase  error.  The  steady 
state  frequency  error  is  equal  to 


While  the  phase  error  increases  as 

Oi  C  01 

n  n 

Thus,  decreasing  £  and  increasing  u>n  reduces  these  errors. 

On  the  other  hand,  it  is  desirable  to  filter  the  effects  of  path 
delay  induced  variations  to  a  level  that  is  roughly  equivalent  to  the 
stability  of  the  reference  clock.  The  most  bothersome  of  these  path 
delay  variations  are  the  daily  variations.  Reduction  of  these  effects 
requries  increasing  £  and/or  decreasing <on>  clearly  in  opposition  to 
requirements  for  tracking  the  reference  or  holding  a  drifting  local 
clock  to  its  reference.  For  a  discussion  of  the  trade-offs  involved  in 
meeting  these  opposing  requirements,  see  Section  3.1.2  of  reference?!**. 
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A  second  criterion  of  tracking  performance  which  was  examined 
was  the  peak  fractional  frequency  error  due  to  a  phase  step. 
Unfortunately,  improvements  in  this  respect  are  at  odds  with  the 
aforementioned  reduction  in  steady-state  phase  error  as  the  peak 
frequency  error  increases  with  increasing  wn_ 

ii.  Acquisition  Time 

It  has  been  shown216  that  a  fast  acquisition  strategy  such 
as  a  two-bandwidth  scheme  cannot  be  employed  with  a  Type  1  loop.  This 
is  due  to  the  fact  that  there  is  no  integrator  to  "remember"  the  nodal 
frequency  while  phase  lock  is  being  achieved.  This  being  the  case,  it 
is  necessary  to  consider  acquisition  performance  as  a  function  of  the 
tracking  parameters.  The  acquisition  time  displays  the  same  functional 
dependence  as  the  phase  error  required  to  track  a  frequency  offset. 

That  is,  it  decreases  with  decreasing  £  and/or  increasing  a>n 

iii.  Stability 

As  was  indicated  earlier,  a  mutual  control  system  can  become 
unstable  and  begin  to  oscillate  under  certain  conditions.  A  lower 
bound  on  £  for  stable  operation  is  £  =  0.707  and  as  this  value  is 
approached,  the  system  exhibits  an  increasingly  oscillatory  transient 
response.  Clearly,  then,  from  a  stability  point  of  view,  larger  values 
of  £  are  desirable. 

iv.  Comparison  With  Type  2  Loop 

For  purposes  of  comparison,  it  was  desired  to  have  the  Type  1 
and  Type  2  loops  perform  as  similarly  as  possible,  given  the  fact  that 
they  are  fundamentally  different.  This  "equivalence  of  performance" 
could  be  measured  by  any  of  a  number  of  criteria.  The  means  that  was 


•  i 


I 


J 

] 


selected  was  to  specify  that  the  frequency  at  which  the  frequency 
response  curve  was  down  20  dB  be  the  same  for  the  two  loops.  Actually, 
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this  required  only  a  very  small  adjustment  in  the  value  of  u>n  chosen 
earlier  216  . 

Sunmarizing  the  trade-offs  involved  in  the  parameter  selection  for  a 
Type  1  loop: 

The  steady-state  phase  error  due  to  a  frequency  step,  the 
steady-state  frequency  error  and  constantly  increasing  phase  error  due 
to  a  drifting  local  clock,  and  the  time  to  acquire  a  frequency  offset 
are  all  decreased  by  decreasing  £  and  increasing  o>n.  The  reduction 
of  £  below  0.707  is  limited  by  network  stability  considerations.  The 
amplitude  of  a  frequency  transient  due  to  a  phase  step  is  reduced  by 
decreasing  both  £  and  o>n.  Limiting  the  effects  of  path  delay 
induced  variations  requires  increasing  £  and  decreasing  o>n. 

With  all  of  the  above  taken  into  consideration,  the  loop  parameters 
selected  for  the  Type  1  loop  were 

£  =  1.0  and  o>n  =  1.52  x  10"3  rad/s. 

The  performance  for  these  parameters  is  sunmarized  below: 

Tracking  Performance  of  Type  1  Loop  With 
£  =  1.0  and  =  1.52  x  10"3  rad/s 
/ 

Fraction 


Input 

Phase  Error 

Frequency  Error 

1  us  Phase  Step 

1  ti s,  peak,  0  st-st 

5.6  x  10"^  peak,  0  st-st 

Fraction  Frequency  Step, 

1.32  ps,  peak 

10-9  peak,  0  st-st 

10-9 

1.32  ps,  st-st 

Natural  Drift  10"3^/day 

Increases  at  0.132  /js 

st-st  =  ^  =  1.523  x  IQ*12  PP 

per  day 

wn 

3. 1.2. 3  Weighting  Coefficients  In  the  Mutual  System 

Another  important  set  of  parameters  are  the  phase  detector  weighting 


coefficients.  Each  node  weights  the  phase  errors  measured  by  a  set  of  relative 
weightings.  Suppose  node  k  has  N  references  available.  Then  each  of  these 


3689A 


3-11 


uoro  no,  in*10  and  1P.57  <is  resriecti ve  1  v.  almost  exactly  one-half  that  at 


references  will  be  assigned  a  relative  weight,  the  weight  from  node  i  being 


denoted  by  w^..  At  any  given  time,  only  some  of  these  references  may  be 
available.  Of  course,  the  actual  weighting  of  those  references  should  reflect 
only  the  active  references.  Assuming  a  subset  of  these  references  are  active, 
then  the  active  nodes  should  be  weighted  by  a  weight  of  the  form 


wi  =  wki^ws’  w*iere 

ws  =  ?  w  ki  ' 

If  the  phase  error  measured  on  link  i  is  denoted  then  the  error  signal 
applied  to  the  loop  fiter  is 

*e  w  s  s  wki  yei. 


For  example,  one  possible  weighting  is  to  weight  all  references  equally.  In  this 
case,  if  there  are  n  active  references,  each  will  be  weighted  by  the  factor  1/n. 
Equal  weighting  has  one  significant  advantage  in  that  it  provides  an  added  measure 
of  survivability.  The  system  frequency  is  not  heavily  influenced  by  a  small 
fraction  of  all  the  nodes.  One  might  try  to  obtain  a  more  stable  network 
frequency  by  more  strongly  weighting  the  most  accurate  clocks  and  the  links  least 
affected  by  path  delay  variations.  This  does  work,  but  the  network  frequency  will 
be  more  strongly  perturbed  by  failures  of  these  more  strongly  weighted  nodes  and 
links.  Determining  the  best  trade  between  these  properties  is  somewhat  subjective 
and  thus  difficult  to  do  analytically. 

The  above  discussion  might  lead  the  reader  to  the  conclusion  that  if 
the  phase  errors  between  the  local  nodal  clock  and  each  input  to  that  node  are 
weighted  equally  in  the  determination  of  the  error  voltage  used  to  steer  the  local 
clock,  all  nodal  clocks  will  contribute  equally  to  the  determination  of  Average 
System  Frequency.  This  is  not  true.  Reference  188  shows  that  when  a  system  is 
made  up  of  3  nodes  and  nodes  B  and  C  each  comnunicate  with  node  A  but  not  with 
each  other  (see  Figure  3. 1.2. 3-1)  the  average  system  frequency  is  a  function  of 
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1  times  the  frequency  of  node  B  plus  1  times  the  frequency  of  node  C  plus  2  times 
the  frequency  of  node  A.  All  the  analysis  performed  in  Reference  188  used  equal 
weighting  of  each  input  phase  error  in  determining  local  clock  error  input. 

Reference  188  also  shows  that  the  Average  System  Frequency  in  the  four 
node  network  shown  in  Figure  3. 1.2. 3-2  is  proportional  to  2  times  Wg  plus  2 
times  W^.  plus  3  times  WA  plus  Wg.  These  apply  to  systems  not  using  Double 
Endedness.  When  Double  Endedness  is  applied  to  the  network  shown  in  Figure 
3. 1.2. 3-1  the  Average  System  Frequency  is  a  function  of  1  times  the  free  running 
frequencies  of  nodes  B  plus  C  plus  4  times  the  free  running  frequency  of  node  A. 

While  it  has  never  been  proven  mathematically,  it  appears  that  in  a 
system  not  using  double  endedness  and  using  Equal  Weighting  the  Average  System 
Frequency  is  a  function  of  the  free  running  frequencies  of  all  the  nodes  weighted 
by  the  number  of  duplex  coimunication  links  connected  to  the  respective  node.  And 
it  appears  that  when  Double  Endedness  is  applied  to  all  links  the  Average  System 
Frequency  is  a  function  of  the  weighted  sum  of  the  free  running  frequencies  of  all 
nodes  with  the  weighting  factors  applied  to  multiple  input  nodes  multipled  by  2  or 
squared  or  something.  It  is  not  clear,  at  this  point,  just  what  the  relationship 
is.  But  the  point  is  that  even  when  Equal  Weighting  is  used  in  averaging  the 
individual  phase  errors  at  a  node  to  determine  the  error  control  to  the  local 
clock,  the  weighted  contribution  a  given  nodal  clock  makes  to  determination  of 
Average  System  Frequency  is  in  some  manner  proportional  to  the  number  of  other 
nodes  communicating  with  it.  Use  of  Double  Endedness  appears  to  further  emphasize 
the  difference  in  influence  on  Average  System  Frequency  contributed  by  nodes  with 
different  connectivity. 

In  this  study.  Simulation  Runs  were  made  using  Equal  Weighting  (EW)  and 
Unequal  Weighting  (UEW).  When  UEW  was  applied,  phase  errors  in  links  from  nodes 
containing  higher  quality  clocks  were  more  heavily  weighted  in  the  averaging  used 
to  determine  the  error  control  input  to  a  nodal  clock.  If  higher  quality  clocks 
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are  used  at  highly  connected  nodes  they  must  already  contribute  more  heavily  to 
the  determination  of  Average  System  Frequency  than  clocks  with  lower 
connectivity.  Use  of  double  endedness  further  emphasizes  a  highly  connected 
node's  influence  on  Average  System  Frequency.  Use  of  increased  weighting  of  the 
links  from  hi gh  quality  clocks  provides  an  additional  factor  which  will  probably 
multiply  the  factors  contributed  by  topology.  Alternately,  if  high  quality  clocks 
are  placed  at  nodes  with  low  connectivity  the  Unequal  Weighting  may  only  serve  to 
make  the  weighting  of  clocks  more  equal  in  determining  Average  System  Frequency. 

Until  the  relationship  between  weighting  of  input  phase  errors  for 
determination  of  error  control  for  the  local  nodal  clock  and  the  influence  of 
topology  on  the  contribution  of  the  free  running  frequency  of  a  node  to  the 
determination  of  Average  System  Frequency  is  better  understood,  the  application  of 
UEW  vs  EW  to  a  system  design  can  only  be  by  guesswork. 

Simulation  runs  were  made  with  both  equal  and  unequal  weighting  so  that 
the  overall  effect  on  the  system  could  be  better  observed. 

3. 1.2.4  Use  of  a  Master 

One  of  the  criticisms  that  has  been  leveled  at  the  mutual  control 
approach  in  the  past  is  that  since  there  is  no  "absolute"  time  (no  master)  it  is 
possible  for  the  overall  system  frequency  to  wander  indefinitely.  This  difficulty 
can  be  alleviated  by  assigning  one  node  to  be  the  "master."  This  node  would  not 
receive  timing  information  from  any  other  node  but  rather  would  only  supply  timing 
to  other  nodes  from  its  (presimably)  very  high  quality  clock.  Factors  to  be 
considered  in  the  selection  of  a  master  node  are  clock  quality,  link  quality, 
topographic  location  in  the  network,  and  probability  of  survival  in  a  stress 
environment  among  others.  Simulations  were  run  both  with  and  without  a  master  to 
determine  its  effect  on  overall  performance. 


3 . 1 . 2. 5  Dropout  Smoothing 
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Another  criticism  of  mutual  control  systems  has  been  that  there  can  be 
large  frequency  transients  at  certain  nodes  in  the  event  of  a  link  dropout.  This 
is  due  to  the  fact  that  when  a  link  drops  out  there  can  be  a  large  step  change  in 
the  average  phase  error  driving  the  loop  at  the  affecteu  node. 

It  has  been  proposed  that  the  magnitude  of  these  frequency  transients 
can  be  greatly  reduced  by  allowing  the  average  phase  error  to  change  gradually 
rather  than  in  a  step.  This  was  accomplished  in  the  simulation  by  noting  the 
phase  error  at  each  node  when  the  link  drops  out  and  then  allowing  the  phase  to 
gradually  shift  to  its  new  value.  Specifically,  an  exponential  delay  is  applied 
to  the  phase  term  from  the  lost  link.  The  effect  of  this  dropout  smoothing  is 
discussed  elsewhere  in  this  report. 

3.1.3  Double-Endedness 

In  a  double-ended  system,  the  nodal  clock  is  corrected  based  on  phase 
error  measurements  made  at  both  ends  of  a  link.  The  end  result  of  this  process  is 
that  the  effect  of  path  delay  variations  is  greatly  reduced.  This  technique  can 
be  implemented  in  both  directed  control  and  mutual  control  systems,  but  in  either 
case,  an  overhead  channel  is  required.  For  a  discussion  of  the  overhead 
requirements  of  this  feature,  see  Paragraph  3.3.7. 

For  reasons  of  clarity,  the  discussion  which  follows  assumes  the  use  of 
a  source  of  "absolute"  time.  This  reference  could  be  provided  via  the  overhead 
channel  and,  if  it  were,  not  only  would  the  path  delay  variation  be  removed,  but 
precise  time  would  be  available  at  each  node.  However,  it  is  possible  to  remove 
the  delay  variation  just  as  well  without  the  knowledge  of  "absolute"  time.  This 
is  accomplished  by  exchanging  the  relative  phase  error  between  the  ends  of  the 
link  as  described  in  section  2.2.2,  rather  than  the  time  error  at  each  end 
relative  to  seme  absolute. 
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Using  a  reference,  this  feature  is  implemented  as  follows: 

Assume  that  we  have  two  nodes,  A  and  B,  and  that  the  link  delays  from  A  to  B  and 
from  B  to  A  are  D^g  and  Dg^,  respectively.  Let  B  transmit  to  A  his  reading  of 
the  absolute  clock,  Tg.  Likewise,  let  A  transmit  to  B  his  reading  of  that  same 
clock,  T^.  When  A  and  B  receive  this  information  they  can  each  measure  the 
difference  between  their  local  clocks  and  the  received  absolute  timing  marks. 
These  differences  are  given  by  (at  A  and  B,  respectively). 


*ta=  -tb-ta 


BA 


and 


4TB  = 


ta  -  tb  -  °AB  ' 


If  each  node  transmits  its  A  T  to  the  other,  then  each  can  calculate 
the  correction  factor 


4tb- 


A  T. 


=  ta  -tb 


It  is  clear  that  if  the  link  is  symmetric  (the  reciprocal  path  delays 
are  equal),  this  quantity  is  simply  the  difference  between  the  two  nodal  clocks, 
regardless  of  path  delay.  Consequently,  corrections  made  in  the  nodal  clocks  will 
be  corrections  in  actual  clock  error  and  not  compensation  for  path  delay 
var i at  ions . 


3.1.4  Independence  of  Clock  Error  Measurement  and  Correction 

Independence  of  clock  error  measurement  and  correction  means  that  a 
node  detecting  an  error  in  its  nodal  clock  in  addition  to  initiating  the  clock 
correction  process,  immediately  transmits  the  uncorrected  clock  error  to  the 
adjacent  clocks  being  disciplined.  This  enables  the  disciplined  clock  to  bias  its 
error  correction  process  such  that  the  error  in  the  preceding  clock  has  no 
influence  on  its  correction.  When  this  procedure  is  employed  throughout  the 
network,  the  result  is  that  all  the  network  clocks  are  actually  tied  directly  to 
the  network  master. 
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Although  one  can  connect  a  master  to  a  mutual  control  system  (See 
Paragraph  3.1.2),  it  does  not  appear  to  be  possible  to  employ  independence  of 
clock  error  measurement  and  correction  with  that  system.  Independence  of  Clock 
Error  Measurement  and  Correction  can  only  be  applied  to  a  system  with  Directed 
Control . 

Let  us  define  the  operation  of  this  feature  more  precisely. 

Suppose  we  have  the  following  network  topology: 


The  arrows  indicate  the  direction  of  timing  information  flow.  Let  the 
local  times  at  nodes  M,  A,  B  and  C,  be  TM,  T^,  Tg,  and  T^.,  respectively. 

Let 

\rAM  •  tm  -  ta 
'tbn  ■  T"  *  Tb 

and  ATcm  =  TM  '  TC 

If  A  transmits  to  B  the  quantities  and  AT^,  then  B  can  calculate  the 
quantity 

ta  +atam  =  ta  '  ta  +  tm  =  V 
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Then  Node  B,  in  turn,  can  calculate  and  transmit  to  Node  C  the  quantities  Tou 

bM 

and  Tg.  This  makes  it  possible  for  C  to  find 

tb  +  atbm  =  tb  '  tb  +  tm  =  V 

Clearly  this  process  can  be  extended  to  all  nodes  in  the  network  with  the  result 
that  each  node  has  direct  access  to  the  master  for  use  in  correcting  its  local 
clock. 


The  quantities  AT^,  etc.,  are  simply  the  measured  but  uncorrected 
clock  errors  with  respect  to  the  master  at  each  node.  For  the  double-ended  system 
discussed  in  the  previous  section,  the  addition  of  this  feature  would  mean 
changing  the  clock  correction  term  to 


+ 


T 


AM 


The  clock  correction  terms  at  the  other  nodes  would  be  modified  similarly. 

3.1.5  Phase  Reference  Combining 

It  has  been  proposed  that  in  a  directed  control  system  employing  both 
double-endedness  and  independence  of  clock  error  measurement  and  correction,  the 
most  significant  performance  degradation  is  due  to  the  variance  in  the  measurement 
of  the  phase  errors.  Phase  reference  comoining  is  a  technique  for  reducing  the 
effect  of  these  measurement  errors.  This  is  accomplished  by  computing  the  phase 
error  with  respect  to  the  master  at  each  node  as  a  weighted  sum  of  the  phase  error 
over  several  paths  to  the  master.  As  in  the  two  previous  sections,  we  will  assume 
that  an  overhead  channel  is  available. 

The  actual  implementation  of  this  feature  is  fairly  complex,  but  the 
underlying  principle  is  quite  simple.  It  is  well  known  that  the  variance  of  the 
average  of  N  independent  random  variables  is  given  by 


where  a  is  the  variance  of  each  of  the  component  random  variables.  It  is 
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2  593  A 

clear,  then  that  if  a  node  computes  its  clock  error  with  respect  to  the  master  as 
an  average  of  errors  computed  over  several  paths  from  the  master  that  the  effects 
of  measurement  errors  can  be  reduced.  The  fact  that  makes  the  implementation  of 
this  feature  complex  is  that  the  multiple  paths  from  the  master  must  be  selected 
so  that  i»  closed  loops  are  formed.  An  algorithm  for  producing  such  a  network  is 
described  in  detail  in  Section  2.2.5. 

Even  greater  reduction  of  measurement  errors  is  obtained  by  the  use  of 
unequal  weightings  in  the  average  when  the  estimates  have  unequal  variances.  Let 
node  X  have  available  to  it  N  distinct  phase  references,  each  having  a  measuranent 
variances  a  (  .  Then  t lie  variance  of  the  "average"  of  these  measurements  is 

minimim  when  each  measuranent  is  weighted  by 


That  is,  each  measuranent  is  weighted  by  the  inverse  of  its  variance  and  the  sun 
then  normalized  by  the  denominator  in  the  above  expression. 

3.1.6  Self-Organizing 

One  of  the  drawbacks  to  the  use  of  a  directed  control  system  which  is 
significant  in  a  military  environment  is  that  the  timing  subsystem  requires 
reorganization  in  the  event  of  a  link  or  node  failure.  When  performed  manual 1y, 
this  can  bo  a  time-consuming  task.  However,  the  method  described  in  Section  2.2.5 
enables  the  network  to  perform  such  adaptation  automat i cal ly?' ^ .  Only  a 
feu  bit/s  of  overhead  channel  capacity  are  required  to  employ  this  approach.  See 
Paragraph  3.3.7  for  a  discussion  of  these  overhead  requirements. 

It  should  be  noted  that  a  self-organizing  feature  has  been  built  into 
the  Canadian  Dataroute198,  93,  103  with  no  difficulty.  Section  6.3  of  this 
report  describes  the  implementation  of  the  synchronization  function  in  the 
Canadian  Dataroute. 
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The  effect  of  this  feature  was  represented  in  the  computer  simulations 
simply  by  shortening  the  time  interval  for  which  the  affected  node  was  without  a 
reference.  In  actual  practice,  it  could  be  expected  that  not  only  would  the 
reorganization  take  place  in  less  time,  but  that  the  new  configuration  would  be 
more  efficient  than  that  produced  by  him  an  operators. 

3.1.7  Illustration  of  the  E ffect  of  Features 

In  sections  3.1.1  through  3.1.6  the  features  that  were  considered  as 
potential  candidates  for  incorporation  into  the  DCS  network  timing/synchronization 
subsystem  were  described  but  words  alone  in  this  case  seem  inadequate  for  getting 
across  to  the  reader  just  what  each  of  these  features  do  in  a  timing/ 
synchronization  subsystem.  Thus,  at  the  risk  of  getting  the  cart  before  the 
horse,  this  section  is  devoted  to  elucidating  upon  the  preceding  descriptions  by 
means  of  graphs  taken  from  the  simulation  results.  However,  those  results  will  be 
described  in  fuller  detail  after  the  simulation  model  has  been  developed.  All  the 
graphs  presenting  results  of  all  simulations  are  grouped  in  Appendix  C.  The 
simulation  approach  will  be  discussed  in  detail  in  Section  3.2.  Section  3. 2. 3. 2 
and  Figure  3. 2. 3.2-1  provide  details  on  the  assumed  Network  Topology.  In  the 
present  section  we  will  illustrate  the  overall  effects  of  the  following  features: 

1.  Mutual  Control 

•  Basi  c  Mutual  Run 

•  Master  in  Mutual  System 

•  Unequal  Reference  Weighting 

•  Double  Ended  Reference  Links 

•  Dropout  Smoothing 

•  All  of  above  versus  none  of  above 


2.  Directed  Control 


•  Basic  Directed  Run 

•  Double-Ended  Reference  Links 

•  Drop- In- Smoothing 

•  Independence  of  Clock  Error  Measurement  and  Correction 

•  Self-Organizing 

•  Phase  Reference  Combining 

•  All  of  above  versus  none  of  above 

3. 1.7.1  Adding  Master  to  Mutual  System  (Figures  C5GP&F  Versus  C3GP&F) 

Figures  C3GP&F  show  the  general  simulation  scenario  under  mutual 
control  and  equal  weighting.  The  following  is  noted. 

1.  All  frequency  and  phase  offsets  were  measured  with  respect  to 
Node  1. 

2.  The  initial  transient  consisted  of  various  frequency  offsets 

g 

ranging  from  0  to  1  part  in  10  . 

3.  Normal  link  variations  were  applied  to  all  links.  These  caused 

g 

peak-to-peak  frequency  variations  of  up  to  1.6  parts  in  10  and 
peak-to-peak  phase  variations  of  up  to  9.3  *<s.  Most  of  this 
variation  was  due  to  the  normal  link  disturbance  on  satellite  Links 
1-13,  1-6  and  6-5. 

4.  Frequency  and  phase  transients  due  to  Link  6-5  dropout  at 
150,000  s,  Link  1-2  dropout  at  200,000  s  and  Node  13 ' s  failure  at 
250,000  s. 

Figures  C5GP&F  show  what  happens  when  Node  1  is  made  the  master  of  the 
network  and  is  not  reciprocally  disciplined  by  other  nodes  of  the  network.  Note 
in  particular  the  following: 

1.  Daily  maximum  peak-to-peak  frequency  variation  reduced  from 
1.6  x  10‘9  to  1.2  x  10“9  with  respect  to  Figures  C3GF. 
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2.  The  frequency  transient  caused  by  failure  of  Link  6-5  at  150,000  s 

increased  Node  5's  transient  from  5  x  10**®  to  8.5  x  10**®  and 

-If)  -Q 

Node  6's  transient  from  6.3  x  10  to  1.2  x  10  . 

3.  The  average  phase  error  of  the  nodes  of  the  network  with  respect  to 
Node  1  seem  to  increase  continuously  throughout  the  run  when 

Node  1  is  made  the  master  node  in  Figure  C5GP. 

3. 1.7. 2  Adding  Unequal  Weighting  to  Mutual  System  (Figures  C4GP&F  Versus  C3GP&F) 

Unequal  weighting  increased  Node  6's  daily  peak-to-peak  frequency 
-9  -9 

variation  from  1.28  x  10  to  1.4  x  10  .  The  unequal  weighting  tends  to 

bring  the  instantaneous  frequencies  closer  to  each  other.  This  is  readily  evident 
from  the  graphs.  However,  the  ensemble  frequency  variance  becomes  greater  when 
unequal  weighting  is  applied.  Application  of  the  unequal  weighting  reduced  the 
total  peak-to-peak  daily  phase  excursions  from  approximately  11.17  ns  to  9.32  ps 
for  a  ratio  of  1.26. 

3. 1.7. 3  The  Double-Ended  Feature  in  the  Mutual  System  (C9GP&F  Versus  C3GP&F ) 
Figures  C9GP&F  show  the  output  of  the  mutual  system  utilizing  the 

double-ended  feature.  As  seen  from  this  figure  the  path  delay  variations  are 
removed  by  this  feature.  As  seen  from  the  phase  plot  a  phase  error  that  is 
proportional  to  the  difference  between  the  local  clock's  natural  frequency  and  the 
reference  frequency  is  required  to  bring  the  local  clock's  frequency  to  the 
reference  frequency.  As  the  local  clock's  natural  frequency  drifts  farther  away 
from  the  reference  frequency  this  phase  error  increases.  The  effects  of  link  and 
nodal  failures  are  clearly  evident  from  Figure  C9GF  at  150,000  s,  200,000  s  and 
250,000  s. 

3. 1.7. 4  Link  Dropout  Smoothing  in  the  Mutual  System  (C7GP&F  Versus  C3GP&F) 
Figures  C7GP&F  show  the  effects  of  link  dropout  smoothing  in  the  mutual 

system.  These  figures,  compared  with  Figures  C3GP&F  show  that  the  frequency 
transients  at  150,000,  200,000  and  250,000  are  smoothed  out. 
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3. 1.7. 5  Mutual  Control  With  Master,  Unequal  Reference  Weiqhtinq  Double-Ended 

and  Dropout  Smoothing  (C10GP&F  Versus  C3GP&F,  C4GP&F.  C5GP&F.  C76P&F 

Figures  C10GP&F  show  the  effects  of  the  application  of  all  features  in 
the  mutual  system.  Comparing  C10GP&F  with  C9GP&F  we  see  that  making  Node  1  the 
master  in  C10GP&F  results  in  a  net  positive  increase  in  phase  errors  throughout 
the  run  with  respect  to  the  master's  phase.  This  indicates  that  even  though  the 
master  is  highly  connected  to  the  network  there  is  a  net  positive  drift  in  natural 
frequencies.  Without  consideration  for  weighting,  the  net  applied  drift  was 
(50-30)  x  10"**  =  20  x  IQ*11  per  day  or  an  average  positive  drift  of  1  part  in 
1011  per  day.  However,  all  ouput  frequencies  remain  tied  to  the  network 
frequency.  This  indicates  that  bit  slips  could  eventually  result  between  the 
master  and  other  nodes  of  the  network.  However,  such  slips  may  be  avoided  by 
occasional  readjustment  of  the  natural  frequency  of  the  drifting  clocks. 

The  failure  of  Node  13  at  250,000  s  resulted  in  a  small  amount  of 
disturbance  of  the  network  frequency  but  this  disturbance  was  considerably  reduced 
from  that  of  the  other  runs  (C3GP&F,  C4GP&F,  C5GP&F,  C7GP&F  and  C9GP&F). 

3. 1.7. 6  Directed  Control 

I  I 

Figures  C2GP  &F  show  the  basic  directed  control  run.  The  daily 
peak-to-peak  frequency  variation  on  Nodes  5  and  11  before  failure  of  Link  6-5  at 
150,000  s  was  1.75  x  10  .  The  corresponding  daily  peak-to-peak  phase 

variations  were  24.7  ps.  These  variations  were  a  result  of  being  disciplined  via 
a  chain  of  two  satellite  links.  Nodes  13,  16  and  17  were  disciplined  through  a 
single  satellite  chain.  They  experienced  daily  peak-to-peak  frequency  and  phase 
variations  with  respect  to  the  master  (Node  1)  of  4.4  x  10"*®  and  6.72  ps 
respectively.  The  reason  why  these  variations  were  closer  to  one-fourth  rather 
than  one-half  of  that  at  Nodes  5  and  11  is  because  the  loop  filter  time  constants 
were  longer  for  the  cesium  clock  at  Node  13  than  for  the  crystal  clock  at  Node  5. 
On  the  other  hand,  the  daily  peak-to-peak  frequency  and  phase  variations  at  Node  6 
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were  8.9  x  10"*®  and  12.57  ns  respectively,  almost  exactly  one-half  that  at 
Nodes  5  and  11.  When  Link  6-5  failed  at  150,000  s.  Node  5  went  back  to  its 
natural  frequency  and  remained  at  this  value  until  160,800  s  and  then  began  to 
reference  Link  7-5.  This  sudden  change  in  phase  error  caused  an  abrupt  change  in 

Q 

Node  5's  frequency  of  approximately  5  parts  in  10  .  Node  11  followed  Node  5 

a 

with  approximately  2  parts  in  10  step  change.  A  similiar  frequency  transient 
occurred  at  Node  16  when  Node  13  failed  at  250,000  s. 

3. 1.7. 7  Orop-In-Smoothinq 

Figures  C2GP&F  show  the  same  run  as  Figures  C2GP  &F  but  with 
coasting  and  drop- in- smoothing  included.  The  most  notable  difference  between 
Figures  C2GP  &F  and  C2GP&F  is  that  the  large  frequency  transients  at  150,000 
and  250,000  s  are  removed  in  Figures  C2GP&F.  The  phase  errors  are  also  affected 
in  that  they  slowly  decay  in  Figure  C2GP.  This  is  to  be  expected  because  the 
integrator  voltage  was  adjusted  to  compensate  for  the  difference  in  phase  errors 
on  the  old  and  new  reference  links  before  and  after  the  failure  respectively. 

3. 1.7. 8  Double  Ended  With  Directed  Control 

I  I 

Figures  C8GP  &F  show  the  effect  of  double  endedness  under  directed 

I  I 

control  when  compared  with  Figures  C2GP  &F  .  It  is  noted  that  the  path  delay 
variations  are  removed  by  this  feature  but  the  large  frequency  transients,  due  to 
phase  error  buildup  without  coasting  during  no  reference,  causes  some  highly 
undesirable  frequency  transients  once  a  new  reference  is  found.  Figure  C8GF  shows 
removal  of  these  undesirable  frequency  transients  when  coasting  and 
drop- in- smoothing  are  applied.  The  transients  at  Nodes  5  and  11  in  Figure  C8GF  at 
150,000  s  and  at  Nodes  16  and  17  at  250,000  s  are  due  to  the  drift  rates  of  the 
clocks  at  these  nodes.  This  drift  rate  requires  a  nonzero  steady  state  phase 
error  that  is  applied  through  the  proportional  path  of  the  Type  2  loop.  As  soon 
as  the  reference  is  lost  this  error  immediately  disappears  and  causes  the 
frequency  steps  shown  on  Figure  C8GF. 
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3.1.  7.9  Independence  of  Clock  Error  Measurement  and  Correction  in  the  Directed 

Control  System 

Figures  C11GP&F  show  the  effect  of  independence  of  clock  error 
measurement  and  correction  when  compared  with  Figures  C8GP  &F  .  In  Figure 
Cl  IGF  the  frequency  transients  at  150,000  s  and  250,000  s  were  due  to  the  clock 
drifts  at  Nodes  5  and  16  respectively.  When  the  new  references  were  found  at 
160,800  s  and  271,600  s  respectively,  smoothing  was  applied  and  Nodes  5  and  16 
returned  to  their  new  reference  frequencies  (the  master)  smoothly.  However,  Nodes 
11  and  17  which  were  referencing  Nodes  5  and  16  respectively  during  the  reference 
outage  were  not  smoothed.  Thus  the  frequency  transients  shown  resulted  at  Nodes 
11  and  17  when  the  master  reference  was  effectively  reacquired  at  160,800  and 
271,600  s  respectively. 

Note  the  effect  of  this  feature  at  50,000  ,  60,000  ,  75,000  and  100,000  s 
when  Figure  C11GF  is  compared  with  Figure  C8GF.  In  Figure  C 1 IGF  nodes  lower  in 
the  hierarchy  are  not  disturbed  by  clock  pertubations  higher  in  the  chain  since 
all  nodes  effectively  are  slaved  to  the  ul  tim  ate  master . 

3.1.7.10  Self  Organizing  in  the  Directed  Control  System 

Figures  C15GP&F  show  the  effect  of  self  organizing  when  compared  with 
Figures  C11GP&F.  In  Figure  C15GF  the  frequency  spike  occurs  at  150,000  s  and 
250,000  s  at  Nodes  5  and  16  due  to  clock  drifts  and  loss  of  Link  6-5  and  Node  13 
at  these  respective  times  but  a  new  reference  is  found  by  means  of  the  self 
organizing  feature  within  a  f ew  minutes  so  no  large  phase  error  builds  up  during 
the  outage  as  in  Figures  C11GP&F.  This  is  practically  the  only  difference  between 
Figures  C15GP&F  and  Figures  C11GP&F. 

The  stress  scenerios  described  in  Section  3.2.3.10  assune  system 
failures.  When  the  self  organizing  feature  is  not  assuned  implemented  the 
sceneros  assune  manual  reorganization  will  be  accomplished  by  site  personnel  and 
require  tens  of  minutes  to  accomplish  it.  When  the  self  organizing  feature  is 
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assuned  to  be  implemented  the  sceneros  assine  the  sane  results  but  being  automated 
it  is  assuned  it  is  accomplished  in  seconds.  Thus  the  only  difference  between  the 
simulations  of  systems  with  and  without  the  self  organizing  feature  is  the  assuned 
time  before  reorganization  is  accomplished  following  a  failure.  It  might  be 
desirable  to  reevaluate  the  time  to  manually  reorganize  a  system  during  a  full 
scale  war  situation. 

3.1.7.11  Phase  Reference  Combining  in  the  Directed  Control  System 

Figures  C16GP*&F*  show  the  effects  of  phase  reference  combining  when 
compared  with  Figures  C15GP*&F*.  In  these  runs  measurement  jitter  terms  were 
applied  as  is  evident  from  the  plots.  The  major  difference  in  the  frequency  plots 
of  Figure  C16GF*  and  Figure  C15GF*  is  at  150,000  s  and  250,000  s  when  Link  5-6  and 
Node  13  fails  respectively.  Node  5  makes  the  initial  frequency  transition  in 
Figure  C15GF*  when  Link  6-5  fails  at  150,000  s  and  then  returns  smoothly  when  a 
new  reference  is  acquired.  This  did  not  happen  in  Figure  C16GF*  because  Node  5 
had  more  than  one  Class  1  reference  with  the  phase  reference  combining  feature. 
Consequently  its  frequency  did  not  change  at  this  time.  At  250,000  s  when  Node  13 
failed,  however.  Node  16  was  left  momentarily  without  a  reference  so  the  frequency 
transient  due  to  drift  at  Node  16  occurred  as  shown  on  both  Figures  C15GF*  and 
C16GF*.  The  return  was  smoothed  in  Figure  C15GF*  but  not  in  Figure  C16GF*  as  no 
snoothing  was  applied  in  the  phase  reference  combining  runs. 

3.1.7.12  Directed  Control  All  Features  Versus  None 

For  contrast  Figures  C16GP*&F*  show  the  effect  of  all  features  versus 

I  I 

none  when  compared  with  Figures  C2GP  &F  . 

3.2  Simulation  Approach 

3.2.1  The  Simulation  Medium 

The  simulations  which  will  be  discussed  in  this  section  were  written  in 
the  GASP  simulation  language.  The  GASP  simulation  language  consists  of  a  set  of 
FORTRAN  IV  subroutines  that  can  perform  many  useful  simulation  functions.  By 
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using  GASP  the  programmer  is  afforded  much  of  the  convenience  of  a  problem 
oriented  language  while  retaining  the  flexibility  and  familiarity  of  the  procedure 
oriented  FORTRAN.  The  simulator  package  utilized  in  this  study  consisted  of  two 
primary  parts  and  one  secondary  part.  The  two  primary  parts  included  a 
preprocessor  and  an  execution  module.  The  secondary  part  consisted  of  a  plotting 
routine.  The  preprocessor  generated  a  disk  file  (or  tape  file)  that  was  used  as 
data  input  to  the  execution  module.  In  this  manner  both  the  preprocessor  and  the 
execution  module  did  not  need  to  reside  in  main  memory  at  the  same  time.  In  turn, 
the  execution  module  produced  a  line  printer  output  as  well  as  a  tape  file  (or 
disk  file).  This  output  file  was  used  as  input  to  the  plotting  routine.  The 
plotting  routine  produced  a  tape  file  which  was  used  as  input  to  a  CALCOMP  plotter 
to  produce  a  plotted  output  of  each  simulation  run.  Part  of  the  line  printer 
output  was  also  in  the  form  of  a  plot.  Statistical  data  was  obtained  from  the 
execution  module  output  file.  All  simulations  were  performed  on  a  Oatacraft 
6024/5  digital  computer. 

3.2.2  The  Simulation  Models 

The  specific  computer  simulation  models  of  the  network  synchronization/ 

timing  subsystems  which  were  used  in  the  simulations  discussed  in  this  report 

were  adapted  from  those  simulation  models  which  were  developed  in  a  previous 
216 

study  conducted  at  Harris  ESD.  The  single  network  topology  depicted  in 

Figure  3. 2. 3.2-1  was  utilized  for  all  simulations.  A  single  topology  allowed  a 
larger  number  of  simulations  to  be  performed  on  the  same  network.  This  provided 
for  a  better  set  of  comparisons  than  would  have  been  the  case  had  the  same  number 
of  simulations  been  spread  out  over  several  network  topologies. 


3-28 


3-16 


2593A 

The  directed  control  feature  was  simulated  by  means  of  the  master-slave 
model  of  the  previous  study.  The  mutual  control  feature  was  simulated  by  means  of 
the  mutual  sync  model  of  that  study.  The  doubled  ended  feature  was  applied  in 
combination  with  both  the  directed  control  and  mutual  control  features.  The 
independence  of  clock  error  measurement  and  correction  feature  was  used  in 
combination  with  directed  control,  double-ended,  phase  reference  combining  and 
self-organizing/nonself-organizing.  The  self-organizing  feature,  as  opposed  to 
nonself-organizi ng  was  implemented  by  utilizing  a  short  time  to  reorganize  through 
reorganization  events  applied  to  the  input  stream  of  the  simulation  program. 
Nonself-organizing  was  implemented  in  the  same  manner  but  the  period  between  a 
change  in  status  of  the  network  and  a  reorganization  was  considerably  longer.  The 
phase  reference  combining  feature  was  simulated  in  conjunction  with  directed 
control ,  double  ended,  and  independence  of  clock  error  measurement  and  correction 
for  both  self-organizing  and  nonself-organizing.  In  order  to  see  the  effects  of 
the  phase  reference  combining,  measurement  error  jitter  terms  were  applied  to  each 
node  and  each  link  for  some  double  ended  simulations  using,  and  not  using,  phase 
reference  combining.  Additionally,  simulations  were  performed  with  the  mutual 
control  feature  utilizing  equal  and  unequal  weightings  as  well  as  link  dropout 
smoothing  and  no  link  dropout  smoothing. 

For  each  combination  of  synchronization  subsystem  features  that  were 
simulated  three  stress  scenarios  were  applied  to  the  network  as  follows: 

•  General  stress  scenario 

•  Low  level  stress  scenario 

t  High  level  stress  scenario 

The  general  stress  scenario  was  designed  to  obtain  a  measure  of  how  closely  nodal 
phases  and  frequencies  at  various  points  of  the  network  would  hold  together  under 
a  variety  of  disturbance  events.  The  low  level  stress  scena*io  was  designed  to 
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measure  any  upward  propagation  of  the  disturbance  to  higher  levels  of  the 
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network.  The  high  level  stress  scenario  was  designed  to  measure  propagation  of 
the  disturbance  in  any  direction,  i.e.,  upward,  laterally  or  downward. 


3.2.3 


Specific  Attributes  of  the  Simulations 


3. 2. 3.1  Loop  Parameters 

With  the  exception  of  the  master  clock  each  nodal  clock  was  disciplined 
by  means  of  a  phase-locked  loop  using  a  linear  phase  detector.  Two  loop  types 
were  employed.  Both  were  second  order  loops  but  they  were  different  in  that  one 
of  the  loop  types  (Type  2)  contained  an  integrator  while  the  other  (Type  1)  did 
not.  To  avoid  problems  of  network  instability  only  the  Type  1  loop  was  utilized 
with  the  mutual  control  feature.  In  order  to  take  advantage  of  the  excellent  long 
term  stability  of  cesium  clocks  nodes  employing  cesium  clocks  were  given  longer 
time  constants  than  those  with  quartz  clocks  under  the  directed  control  feature. 
Table  3. 2. 3.1  shows  the  loop  parameters. 

3. 2. 3.2  Network  Topology 

Figure  3. 2. 3. 2-1  shows  the  network  topology  used  for  all  simulations. 
This  topology  was  chosen  to  somewhat  resemble  a  skeletal  representation  of  the  DCS 
network  in  that  distances  were  chosen  to  approximate  North  American,  transatlantic, 
and  European  distances. 

3. 2. 3. 3  Network  Hierarchy 

Three  levels  of  hierarchy  were  defined  and  adhered  to  where  applicable 
in  all  simulations.  The  hierarchy  was  used  primarily  as  an  index  of  clock  and 


link  qualities.  The  hierarchy  was  as  follows: 


Nodes 


1,  2,  13,  and  14 

3,  4,  5,  6,  7,  8,  15,  and  16 

9,  10,  11,  12,  17,  18,  19,  and  20 
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Table  3. 2. 3.1.  Loop  Parameters 
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•  Directed  Control  Quartz  Clocks  -  Type  2  Loop 

f  =  4,  «n  =  5.6  x  IQ'5  rad/s 

•  Directed  Control  Cesium  Clocks  -  Type  2  Loop 

f  =  2,  o>n  =  1.12  x  10~5  rad/s 

•  Mutual  Control  and  Directed  Control  (All  Clocks)  -  Type  1  Loop 

£  =  1.  wn  =  1.52  x  10"^  rad/s 

These  parameters  were  chosen  to  make  the  closed  loop  transfer  function 
provide  roughly  the  same  jitter  attenuation  as  the  directed  control 
Type  2  loop. 


3. 2. 3. 4  Link  Type  and  Distance 

Table  3. 2. 3. 4  shows  link  types  and  distances  used  in  all  simulations. 

3. 2. 3. 5  Initial  Directed  Control  Tree 

Initially,  and  after  each  node  or  link  failure,  a  unique  tree  described 
the  paths  for  dissemination  of  reference  information  for  each  directed  control 
simulation.  Figure  3. 2. 3. 5-1  shows  the  initial  directed  control  tree.  The 
reorganization  events  successively  modify  this  tree  to  reflect  the  new 
organizati  on. 


3689A 


3-19 


Table 

Links  Type  Distance 

1  -  2 
1  -  6 
1  -  7 
1  -  8 
1  -  13 

1  -  14 
2-3 

2  -  4 
2-6 

2  -  13 

3  -  4 

3  -  5 

4  -  5 

5  -  6 
5-7 
5  -  11 
5  -  12 

7- 8 
8  -  9 

8- 10 
13  -  15 

13  -  16 

14  -  15 
14  -  19 

14  -  20 

15  -  16 

16  -  17 
16  -  18 
19  -  20 


3. 2. 3.4.  Link  Type  and  Di 


Land  Cabl  e 

Satel 1 i te 

Land  Cabl e 

Microwave 

Satellite 

Submari ne  Cable 

Microwave 

Land  Cable 

Microwave 

Submari  ne  Cable 

Microwave 

Microwave 

Microwave 

Satel lite 

Land  Cable 

Mi crowave 

Microwave 

Land  Cable 

Microwave 

Microwave 

Microwave 

Tropo 

Tropo 

Microwave 

Tropo 

Tropo 

Microwave 

Mi crowave 

Microwave 
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Initial  Transient 
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An  initial  transient  was  applied  to  each  simulation  run.  The  initial 
transient  consisted  of  a  specific  set  of  frequency  offsets  from  some  absolute 
value.  Table  3. 2. 3. 6  shows  the  initial  offsets  that  were  applied  to  each  node  for 
all  simul ati on  runs . 

3. 2. 3. 7  Clock  Drifts 

A  natural  frequency  drift  was  applied  to  all  quartz  clocks  for  the 
duration  of  each  simulation  run.  Table  3. 2. 3.7  shows  these  drift  terms. 

3. 2. 3. 8  Link  and  Nodal  Measurement  Jitter 

In  order  to  evaluate  the  utility  of  the  phase  reference  combining 
feature,  measurement  jitter  terms  were  included  in  each  link  and  node  as  shown  in 
Table  3. 2. 3.8.  Class  1  and  Class  2  nodal  output  variances  and  nodal  weighing 
factors  were  then  calculated  in  accordance  with  these  measurement  jitters  and 
applicable  directed  control  tree.  The  calculations  proceeded  in  stages  as 
follows:  1)  Calculate  Class  2  output  variance  for  master  node.  2)  Calculate 
Class  1  output  variances  and  weighing  factors  for  all  nodes  that  are  one  node 
below  the  network  master.  3)  Calculate  Class  2  output  variances  and  weighing 
factors  for  all  nodes  that  are  one  node  below  the  master.  4)  Repeat  the 
calculations  of  2)  and  3)  for  all  nodes  that  are  two  nodes  below  the  master, 
three  nodes  below  the  master,  etc.  At  Node  K  the  output  variances  were  calculated 
according  to  the  following  formula: 
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Node  Initial  Frequency  Offset 

1  fo 

2  fo+10“Hxfo 

3  fo  -  10-H  x  fo 

4  fo  +  5  x  10-H  x  f0 

5  fo  +  2  x  10"11  x  f0 

6  fo  -  10-10  x  fo 

7  fo  +  3  x  10-H  x  f0 

8  fo  +  7  x  10-H  x  f0 

9  fo  -  10-0  x  f0 

10  fo  +  10-9  x  fo 

11  fo  +  5  x  10-10  x  f0 

12  fo  +  2  x  10-10  x  f0 

13  fo  -  10-H  x  fo 

14  fo  -  5  x  10-12  x  f0 

15  fo  +  2  x  10“11  x  f0 

16  fo  -  3  x  10-11  x  f0 

17  fo  +  10-9  x  fo 

18  fo  +  10-9  x  fo 

19  fo  -  5  x  10-10  f0 

fo  -  2  x  10-10  x  f0 


20 


Table  3.2. 3. 7.  Nodal  Clock  Drifts 

Daily  Drift 
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Table  3. 2.  3. 8.  Measurement  Jitter  Variances 
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Tropo 
Land  Cable 


Submarine  Cable 


Nodes 

1,  2,  13  and  14 

3,  4,  5,  6,  7,  8,  15,  and  16 

9,  10,  11,  12,  17,  18,  19,  and  20 


1  <J  Measurement  Jitter  Variance 
30  ns 
10  ns 
20  ns 
10  ns 
10  ns 

10  ns 
20  ns 
30  ns 


Individual  jitter  variances  are  from  independent  normally  distributed  random 
variates  with  zero  mean  and  one  standard  deviation  equal  to  the  above  values. 
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Each  of  these  terms  is  explained  in  Paragraph  3.3.11.4.2.  Similarly  the  weighing 
factors  to  be  applied  to  the  i^  reference  term  were  determined  by  the  following 
f  ormul  a: 
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3. 2. 3.9  Normal  Link  Variations 
A  set  of  normal  link  variations  were  applied  to  all  links  for  the 

general  stress  scenario  and  to  a  single  link  for  the  high  and  low  level  stress 
scenario  runs.  These  normal  link  variations  were  chosen  according  to  link  type 

i 

and  are  shown  in  Table  3. 2. 3. 9. 

3.2.3.10  Stress  Scenarios 

The  general,  low  level,  and  high  level  stress  scenarios  are  given  in 
Tables  3.2.3.10-1,  3.2.3.10-2,  and  3.2.3.10-3  respectively. 

3.2.3.11  Feature  Combinations  for  Simulations 
Table  3.2.3.11  shows  the  combinations  of  features  that  were  used  in  the 

simulations. 

3.3  Analysis  of  the  Features  Against  Desirable  Characteristics 
This  Section  specifically  addresses  each  of  the  twelve  "Desirable 

Characteristics  of  a  Digital  Communications  Timing  Subsystem"  presented  in  Section 

2.3  and  attempts  to  evaluate  "as  quantitatively  as  possible"  to  what  degree  each 
of  the  timing  subsystem  feature  combinations  contribute  to  improvement  in  each 
characteristic. 
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Table  3. 2. 3. 9.  Normal  Link  Variations 


Link  Type 

Microwave 

AL  =  10-5  x  L0  x  sin  ((u<jt  +4>r) 

Cable 

AL  =  3  x  10-6  x  Lo  x  sin  if*>d t  +*i>r) 

Satellite 

AL  =  5.04  x  10-5  x  Lq  x  sin  (<^dt  +  <f>r) 

*Tropo 

AL  =  1  x  10-8  x  L0  x  sin  (wxt  +  </> r) 

Where:  <f>r  =  q 

“d  =  7.2685  x  10-5  rad/s 
(jJx  =  3.5  x  10-3  rad/s 
Lo  =  nominal  link  distance 

The  maximum  values  of  distance  variation  can  be  expressed  as  a  maximum  time 
variation  by  the  following  formula: 


Where:  K  is  the  numerical  constant  given  for  each  link  type  and 

v  is  the  nominal  speed  of  transmission  along  the  applicable  link  type. 

*  See  Paragraph  3. 3. 4. 5. 
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Table  3.2.3.10-1.  General  Stress  Scenario 
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•  Apply  initial  transient  to  all  nodes 

•  Apply  normal  link  delay  variations  to  all  links 

•  Apply  clock  drifts  to  all  quartz  clocks 

50,000  s  -  Node  6  clock  starts  ramp  increase  in  frequency  of  1.16  x  10”^  x 
fo  per  second  and  continues  until  75,000  seconds. 

60,000  s  -  Node  13' s  clock  makes  step  decrease  in  natural  frequency  of  3  x 
10-H  x  f0> 

75,000  s  -  Node  6's  clock  begins  ramp  decrease  of  1.16  x  10-14  x  fo  per 
second  and  continues  until  100,000  seconds. 

100,000  s  -  Node  13 's  clock  makes  step  increase  of  3  x  10"H  x  fo  in  natural 
frequency. 

150,000  s  -  Link  6-5  fails. 

Node  5  free  runs  until  150,300  seconds  and  then  references  Node  7  if 
self-reorganizing. 

Node  5  free  runs  until  160,800  seconds  if  nonself-reorganizing. 
200,000  s  -  Link  1-2  fails. 

Node  2  free  runs  until  200,300  seconds  it  references  Node  6  if  using 
self-reorganization,  it  free  runs  to  210,800  seconds  for  nonself¬ 
reorganization. 

250,000  s  -  Node  13  fails. 

Node  16  free  runs  until  250,300  seconds  at  which  time  it  references 
Node  15  if  using  self-reorganization. 

It  free  runs  to  271,600  seconds  if  using  nonself-reorganization. 

Monitor:  Nodes  5,  6,  7,  2,  11,  13,  16,  17,  15,  and  3 

Links  7-5,  1-6,  1-7,  6-2,  5-11,  1-13,  13-16,  16-17,  14-15,  and  2-3. 

Note:  When  the  self  reorganizing  feature  is  not  assimed  implemented  this 

scenerio  assumes  manual  reorganization  will  be  accomplished  by  site 
personnel  in  about  3  hours.  When  the  self  reorganizing  feature  is 
assimed  implemented,  it  is  assumed  that  the  same  new  organization 
results  but  is  accomplished  automatically  in  about  5  minutes.  In  a 
wartime  environment  the  time  to  manually  reorganize  might  deserve 
further  consideration. 
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Table  3.2.3.10-2.  Low  Level  Stress  Scenario 

•  Apply  initial  transient  to  all  nodes 

•  Apply  clock  drifts  to  all  quartz  clocks 

•  Apply  link  variations  on  links  6-5 
100,000  s  -  Links  6-5  fails. 

Node  5  then  free  runs  until  100,300  seconds  for  self-reorganization 
and  then  references  Node  7. 

Node  5  free  runs  until  107,200  seconds  and  then  references  Node  7  if 
nonself-reorganizi  ng. 

150,000  s  -  Node  11  fail 

200,000  s  -  Node  3  fails. 

250,000  s  -  Step  change  of  +10~9  x  fo  in  Node  5's  natural  clock  frequency. 
Monitor:  Nodes  -  12,  7,  5,  4,  3,  2,  6,  8,  13,  and  14 

Links  -  5-12,  1-7,  7-5,  2-4,  2-3,  1-2,  1-6,  1-8,  2-13,  and  1-14 
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Table  3.2.3.10-3.  High  Level  Stress  Scenario 
Apply  initial  transient 
Apply  clock  drifts  to  all  quartz  clocks 

Measure  phase  WRT  Node  1  until  Node  5  fails,  thf  :  measure  all  phase  WRT  Node  2 
Apply  path  delay  variation  on  link  1-13 
50,000  s  -  Clock  at  Node  2  up  by  +10*10  x  f0  natural  frequency 
100,000  s  -  Clock  at  Node  2  down  by  -10*10  x  fo  natrual  frequeuncy 
150,000  s  -  Link  1-13  fails 

Node  13  free  runs  until  150,300  seconds  and  then  references  Node  2  if 
self -reorganization. 

Node  13  free  runs  until  157,200  seconds  and  then  references  Node  2  if 
nonself -reorganizing 

200,000  s  -  Node  1  fails 

Nodes  2,  6,  7,  8,  and  14  free  runs  until  the  following  times 
Self-Reorganizing 

200,300  s  -  Node  2  becomes  new  master 
Node  6  references  Node  2 
Node  15  references  Node  13 
Node  14  references  Node  15 
Node  7  references  Node  5 
Node  8  references  Node  7 
Nonself -Reorganizing 

203,600  -  Node  2  becomes  the  new  master 

Node  6  references  Node  2 

Node  15  references  Node  13 

Node  14  references  Node  15 

Node  7  references  Node  5 

Node  8  references  Node  7 

Nodes  -  2,  3,  5,  6,  14,  15,  16,  17,  11,  13 

Links  1-2,  2-3,  6-5,  2-6,  15-14,  13-16,  16-17,  5-11, 
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200,600  s 
200,900  s 
201,200  s 


203,600  - 

207,200  - 
210,800  - 
214,400  - 
Monitor  - 
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Table  3.2. 

3.11.  Feature  Combinations  for  Simulations 

1. 

DC-1 

Directed  control  with  Type  1  loop  (mutual  sync  loop 
parameters) . 

2. 

DC-2 

Directed  control  with  Type  2  loop. 

3. 

MC+EW 

Mutual  control  with  equal  weighting. 

4. 

MC+UEW 

Mutual  control  with  unequal  weighting. 

5. 

MC+M+EW 

Mutual  control  with  a  master  and  equal  weighting. 

6. 

MC+M+UEW 

Mutual  control  with  a  master  and  unequal  weighting. 

7. 

MC+EW+DOS 

Mutual  control  with  dropout  smoothing  (and  equal 
weighting). 

8. 

DC+DE 

Directed  control  with  Type  2  loop  and  double-ended. 

• 

9. 

MC+DE+EW 

Mutual  control  with  equal  weiqhtinq  and  double-ended. 

10. 

MC+W+DE+UEW+DOS 

Mutual  control  with  a  master,  unequal  weighting,  dropout 
smoothing,  and  double-ended. 

[ 

11. 

DC+DE+ICEM&C 

Directed  control  with  double-ended  and  independence  of 
measurement  and  correction. 

12. 

DC+DE+ICEMiOPRC 

Directed  control  with  double-ended,  independence  of 
measurement  and  correction,  and  phase  reference 

■ 

combining. 

SELF-ORGANIZING  RUNS 

13. 

DC+SO 

Directed  control  with  Type  2. 

14. 

DC+DE+SO 

Directed  control  with  double-ended. 

15. 

DC+DE+I CEM&C+SO 

Repeat  Run  No.  11. 

16. 

DC+DE+ ICEM&C+PRC+SO 

Repeat  Run  No.  12. 
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The  First  six  characteristics  were  evaluated  through  the  measurement  of 
system  performance  using  the  results  of  computer  simulations.  The  renaining  six 
characteristics  utilize  other  forms  of  analysis  to  evaluate  system  performance. 

3.3.1  Maximize  Frequency  Accuracy 

The  general  simulation  scenario  runs  have  been  used  to  evaluate  this 
desirable  characteristic.  Data  has  been  tabulated  in  the  form  of  ensemble 
averages  over  the  monitored  nodes.  These  ensemble  averages  were  made  on  the 
following  quantities: 

1.  Magnitude  of  mean  nodal  frequency  error  over  the  simulation  run 
durati on 

2.  Variance  of  nodal  frequency  error  over  the  simulation  run  duration 

3.  Magnitude  of  mean  nodal  phase  error  over  the  simulation  run  duration 

4.  Variance  of  nodal  phase  error  over  the  simulation  run  duration 
Ratios  of  these  ensemble  averages  were  then  taken  for  various  feature 

combinations  within  the  directed  control  regime  and  within  the  mutual  control 
regime.  Finally,  ratios  were  taken  with  various  feature  conbi nati ons  between  the 
mutual  control  regime  and  the  directed  control  regime. 

3. 3. 1.1  The  Ensemble  Averages 

Table  3. 3. 1.1  shows  the  ensemble  averages  of  frequency  and  phase  error 
means  and  variances  for  the  general  simulation  scenario  runs.  The  most  significant 
aspect  of  this  table  are  the  averages  over  all  runs.  These  averages  give  the 
system  designer  a  feel  for  the  sizes  of  frequency  and  phase  errors  that  may  be 
expected  to  be  found  in  the  network.  The  most  striking  comparison  that  one  can 
make  between  the  feature  combinations  of  this  table  is  that  of  double  ended  versus 


Table  3. 3. 1.1.  Ensemble  Frequency  and 


<  in  A/  |> 

Feature 

Proportional 

Combinat.  i  on 

Parts 

1G  OC-1 

6.244 

x  10-12 

2G  OC-2 

7.29 

x  10*12 

2G '  OC-2 

4.56 

x  10-12 

3G  MC+EW 

2.382 

x  10-H 

4G  MC+UEW 

2.335 

x  10-H 

5G  MC+M+EW 

8.233 

x  10-12 

6G  MC+M+UEW 

2.91 

x  10-12 

7G  MC+EW+DOS 

3.92 

x  10-H 

8G  DC+DE 

9.67 

x  10-13 

9G  MC+DE+EW 

4.45 

x  10-H 

10G  MC+M+DE+UEW+DOS 

4.93 

x  10-12 

11G  DC+DE+ICEM&C 

1.189 

x  10-12 

11G*  OC+DE+ICEM&C 

8.763 

x  10-13 

12G*  DC+DE+ICEM&C+PRC 

4.97 

x  10-13 

13G  DC+SO 

8.21 

x  10-12 

14G  OC+DE+SO 

6.284 

x  10-13 

15G  DC+DE+ICEM&C+SO 

4.953 

x  10-13 

15G*  DC+DE+ICEM&C+SO 

5.729 

x  10-13 

16G*  DC+DE+ICEM&C+PRC+SO 

4.97 

x  10-13 

Phase  Averages  for 

General  Simulations 

<ff2 A/ > 
(Proportional 
_  Parts)? 

2 

<  |  |i  \</>|Xir  \</>  s 

_ 

2.689 

X 

10-10 

0.455 

3.406 

1.822 

X 

10-10 

1.237 

3.007 

1.969 

X 

10-10 

0.374 

2.606 

3.034 

X 

io-io 

1.431 

2.484 

3.891 

X 

io-io 

1.011 

2.351 

1.594 

X 

io-io 

1.556 

2.450 

1.625 

X 

io-io 

0.673 

2.29 

3.09 

X 

io-io 

1.350 

2.43 

1.481 

X 

lo-n 

0.255 

0.239 

1.68 

X 

lo-n 

0.951 

0.348 

1.31 

X 

10-11 

0.728 

'  0.273 

1.433 

X 

10-11 

0.270 

0.184 

1.509 

X 

10-11 

0.270 

0.183 

1.464 

X 

10-11 

0.242 

0.141 

1.842 

X 

10-10 

1.267 

3.111 

1.39 

X 

10-11 

0.250 

0.170 

1.195 

X 

10-11 

0.238 

0.164 

1.32 

X 

10-11 

0.246 

0.136 

1.279 

X 

10-11 

0.238 

0.129 

♦With  Jitter 
'Without  Smoothing 


3-46 


2593  A 

single  ended  reference  links.  Averaging  the  ensemble  averages  over  all  runs  with 


single  ended  links  (1G  through  7G  plus  13G)  and  with  double  ended  links  (3G 
through  16G  excluding  13G)  and  then  taking  ratios  we  find  the  following: 


<  1  P  a/  1  > 

<  cr  A /> 

'  l 

2 

<<r  Ap> 

Single 

Ended 

Mean 

1.491  x  10'U 

2.45  x  10-10 

1.129 

2.691 

Double 

Ended 

Mean 

6.71  x  10-12 

1.40  x  10-H 

0.397 

0.206 

Ratio 
of  Single 
Ended  to 
Double 

Ended 

Means 

2.22 

17.44 

2.831 

13.064 

However,  it  should  be  kept  in  mind  that  quantities  which  can  take  on 
both  positive  and  negative  values  during  the  run  such  as  frequency  error  and  phase 
error  will  tend  to  have  a  time  average  which  is  somewhat  dependent  on  run  duration 
if  the  run  duration  is  not  an  exact  multiple  of  all  the  periods  of  the  link 
vari  ati  ons . 

3.3.1.?  Comparisons  Within  the  Directed  Control  Regime 

Table  3. 3. 1.2  shows  the  ratios  of  frequency  and  phase  ensanble  averages 
for  several  runs  within  the  directed  control  regime.  A  suprising  comparison 
occurs  between  2G  and  13G.  This  ratio  indicates  better  performance  for  the 
non-self  organizing  run  of  2G.  However  this  is  deceiving  since  this  does  not  hold 
in  general.  Further  scrutiny  of  the  nodal  averages  shows  that  the  only  nodes  for 
which  Run  2G  showed  smaller  errors  than  Run  13G  was  at  Nodes  2  and  3.  Then  a  look 
at  the  plots  of  Figures  C2GP&F  and  C13GP&F  shows  that  these  nodes  had  identical 
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Table  3. 3. 1.2  Comparisons  Within  Directed  Control  Runs 


Feature 

Combination 

Ratio 

Ratio 

of 

<|PA/|> 

Ratio 

of 

<jd£ /> 

Ratio 

of 

<  1  // A0|> 

Ratio 

of 

<  cr  ?A  0> 

?/R  DC-2 

2/8  UC+DF 

7.54 

12.3 

4.85 

12.58 

2/13  15C+50 

0.888 

0.99 

0.976 

0.966 

DC-2 

2/14  DC+5F+S0 

11.6 

13.11 

4.95 

17.69 

o/ii  DC-2 

2/11  DC+bE+ICEM&C 

6.13 

12.71 

4.58 

16.34 

o  /i  c  DC-2 

2/15  BT+DFUCEM&C+SO 

14.72 

15.25 

5.20 

18.34 

q  /i  i  DC+DE 

8/11  urwumu 

0.813 

1.033 

0.944 

1.299 

o/ii  DC+DE 

8/14  T5F+BF+5& 

1.54 

1.07 

1.02 

1.41 

q / ic  DC+DE 

8/15  T5C+DE+ICEM&C+S0 

1.952 

1.239 

1.071 

1.457 

,i*,„*  DC+DE+ICEM&C 

11  /1Z  DC+DE+ICEM&C+PRC 

1.763 

1.031 

1.116 

1.298 

ic*/iC*  DC+DE+ICEM&C+SO 

15  /16  DC+DE+ICEM&C+PRC+SO 

1.153 

1.032 

1.034 

1.054 

DC-2 

2/16*  BC+UE+ICEM&C+PRC+SO 

14.67 

14.25 

5.20 

23.31 

1/2  [jf~l 

V/L  urc? 

0.857 

1.476 

0.368 

1.133 

2+8+11 

13+14+15 

1.01 

1.01 

1.004 

0.996 

11++15* 

12*+16* 

1.458 

1.031 

1.075 

1.181 

11*/11 

0.74 

1.05 

1.00 

0.99 

15*/15 

1.16 

1.11 

1.03 

0.83 
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errors  until  Link  1-2  failed  at  200,000  seconds.  A  new  reference  was  then 
established  for  Node  2  via  Link  6-2.  Node  6  obtained  its  reference  via  Link  1-6, 
a  satellite  link.  In  Run  13G  Node  2  established  the  new  reference  10,500  seconds 
before  it  did  in  Run  2G.  Thus  it  turned  out  that  self  reference  for  Node  2  during 
the  interval  200,300  to  210,800  was  better  than  the  varying  reference  via  the 
satellite  link  during  this  same  interval. 

Comparison  2/8  shows  the  large  performance  improvement  for  double  ended 
versus  single  ended  reference  links.  Comparison  2/14  shows  the  improvement  of 
double  ended  and  automatic  reorganization  over  single  ended  and  manual 
reorganization.  Comparison  2/11  shows  the  improvement  of  double  ended  and 
independence  of  clock  error  measurement  and  correction  with  respect  to  single 
ended.  Comparison  8/11  shows  the  relative  performance  of  double  ended  and  double 
ended  with  independence  of  clock  error  measurement  and  correction.  Adding  the 
ICEM&C  feature  increased  the  ensemble  of  magnitude  of  mean  frequency  and  phase 
errors  but  decreased  the  ensemble  average  of  frequency  and  phase  error  variances. 
Scrutiny  of  nodal  averages  reveals  that  the  only  nodes  for  which  Run  11G  showed 
larger  average  errors  were  at  Nodes  5,  11,  and  17  but  primarly  at  11  and  17.  A 
look  at  the  plots  then  shows  the  times  at  which  these  errors  were  larger.  The 
larger  errors  occurred  immediately  after  150,000  and  250,000  seconds  when  Link  6-5 
and  Node  13  failed  respecti vely.  As  previously  mentioned  smoothing  was  applied  to 
only  one  node  below  the  point  at  which  the  disturbance  occurred.  This  resulted  in 
a  significant  frequency  spike  when  the  master  was  reacquired  in  Run  11G  at  nodes 
11  and  17  due  to  the  frequency  drift  at  these  nodes.  This  is  the  reason  that  Run 
11G  shows  larger  ensemble  of  magnitude  of  frequency  and  phase  error  means  than  did 
Run  8G.  It  is  further  noted  that  this  is  not  intrinsic  to  the  independence  of 
clock  error  measurement  and  correction  feature  but  rather  a  quirk  of  the 
implementation  of  the  simulation  which  was  used.  Comparison  8/14  shows  the 
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Improvement  that  automatic  reorganization  has  for  a  double  ended  system. 

Comparison  8/15  shows  the  improvement  obtained  by  adding  automatic  reorganization 
and  independence  of  clock  error  measurement  and  correction  to  a  double  ended 
system.  Comparison  11*/12*  shows  the  improvement  that  phase  reference  combining 
provides  to  the  double  ended  system  with  independence  of  clock  error  measurement 
and  correction.  Comparison  15*/16*  shows  the  improvement  that  phase  reference 
combining  provides  to  a  double  ended  system  with  automatic  reorganization  and 
independence  of  clock  error  measurement  and  correction. 

3. 3. 1.3  Comparisons  Within  the  Mutual  Control  Regime 

Table  3. 3. 1.3  shows  several  comparisons  within  the  mutual  control 
runs.  Comparison  3/4  shows  the  effect  of  unequal  weighting  versus  equal 
weighting.  There  were  slight  improvements  in  mean  frequency  error,  mean  phase 
error  and  phase  variance.  There  was  a  degradation  in  the  frequency  variance  ratio 
of  0.78.  This  degradation  showed  up  at  approximately  this  same  ratio  at  all 
monitored  nodes.  Comparison  3/5  shows  the  effect  of  making  Node  1  of  the  network 
the  master.  In  this  case  Node  1  Influences  the  frequency  of  all  of  its  neighbor 
nodes  but  its  own  frequency  is  not  influenced  by  any  of  its  neighbors.  This  tends 
to  make  the  results  of  mean  frequency  and  phase  errors  dependent  on  the  run 
duration  if  this  duration  is  not  an  exact  integer  multiple  of  all  the  periodic 
link  disturbances.  Since  variances  are  not  influenced  by  the  sign  of  quantities 
being  measured  they  are  not  dependent  on  the  relationship  between  run  duration  and 
the  period  of  link  disturbances.  Making  Node  1  the  network  master  reduced  the 
frequency  variance  by  a  factor  of  1.9  and  left  the  phase  variance  almost  unchanged 
at  1.01. 
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Table 

3. 3. 1.3. 

Comparisons  Wlthlr 

i  Mutual  Control 

Runs 

Feature 

Ratio 

Ratio 

Ratio 

Ratio 

Combination 

of 

of 

of 

of 

Ratio 

lLt£Al!> 

<<r  2  A  f> 

<luA0l> 

<o-2a<7>> 

3/4  MC+EW 

MC+UfcW 

1.02 

0.78 

1.42 

1.06 

,/e  MC+EW 

3/5  HC+H+EW 

2.89 

1.90 

0.92 

1.01 

, /e  MC+EW 

3/6  MC+M+OEW 

8.19 

1.87 

2.13 

1.08 

,  ,c  MC+UEW 

4/6  MC+M+UEW 

8.02 

2.39 

1.50 

1.03 

**  n  MC+EW 

3/7  MC+EV+DOS 

0.61 

0.98 

1.06 

1.02 

o  /«  MC+EW 

3/9  MC+DE'+EM 

0.54 

18.06 

1.50 

7.14 

o/ in  DC+DE+EW 

9/10  MC+H+UE+UEW+DDS 

9.90 

1.28 

1.31 

1.27 

MC+EW 

3/10  HC+M+UE+UEU+DU5 

4.83 

23.16 

1.97 

9.10 

3+4+5+6+7 
~ 9+IU - 

0.79 

17.76 

1.43 

7.73 

3+4+7 +9 

5+6+10 

6.10 

2.28 

1.20 

1.14 

3+5+7+9 

~7T5TTTT 

2.78 

1.05 

1.64 

1.18 

4+4+5+6+9 
~7TW - 

0.93 

1.28 

1.08 

1.47 

3+4+5+7+9 
“PID - 

7.10 

2.68 

1.80 

1.57 
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Comparison  3/6  shows  the  effect  of  a  master  and  unequal  reference 
weighting.  When  compared  with  comparison  3/5  there  was  an  improvement  factor  of 
2.83  in  frequency  error  and  an  Improvement  factor  of  2.J2  In  phase  error. 

Comparison  3/7  shows  the  effect  of  dropout  smoothing.  The  most 
significant  effect  was  that  the  average  frequency  error  was  degraded  by  a  factor 
of  0.61.  This  represents  a  penalty  in  order  to  reduce  frequency  spikes  to 
neglegible  values.  The  other  parameters  remained  almost  unchanged. 

Comparison  3/9  shows  the  effect  of  double  endedness  on  the  statistical 
averages.  As  expected  this  feature  reduced  the  ensemble  of  frequency  and  phase 
variances  by  considerable  amounts,  18.06  and  7.14  respecti vely.  However,  the 
surprising  effect  was  the  degradation  in  frequency  error  by  a  factor  of  0.54.  A 
look  at  the  average  errors  at  the  individual  nodes  showed  degradation  at  all  nodes 
except  Node  13  which  was  improved  by  about  50  percent.  Thus  the  degradation  was 
spread  over  almost  all  of  the  network  with  little  deviation  from  node  to  node.  If 
one  looks  at  the  plots  he  sees  that  the  run  duration  was  approximately  3.5  periods 
of  the  daily  satellite  link  variations.  Thus  there  was  a  surplus  of  one  negative 
half  cycle  of  the  link  variations  which  tended  to  cancel  part  of  the  net  positive 
initial  transient  and  net  positive  average  drift. 

Comparison  3/10  shows  the  effects  of  all  additional  features  for  mutual 
control  versus  none  of  the  additional  features.  The  most  striking  performance 
improvement  is  realized  in  reduction  of  frequency  error  variance  by  a  factor  of 
23.16.  Similarly  the  phase  error  variance  was  reduced  by  a  factor  of  9.1. 

Comparison  9/10  shows  the  effect  of  all  features  except  double 
endedness.  The  frequency  and  phase  error  variances  were  reduced  by  factors  of 
1.28  and  1.27  respecti vely. 
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■1.3. l.A  Comparisons  Between  Mutual  and  Directed  Control  Runs 

Table  3. 3. 1.4  shows  comparisons  for  several  pairs  of  runs  in  the  mutual 
regime  versus  the  directed  regime.  Comparison  3/2  shows  the  basic  mutual  to 
directed  ratio.  The  directed  regime  seems  to  provide  some  improvement  in  all 
parameters  except  phase  variance.  The  individual  nodal  statistics  show  that  the 
mutual  system  had  smaller  variances  on  the  left  hand  side  of  the  network  (2, 5,6, 7, 
and  11)  while  the  directed  system  had  the  smaller  variances  on  the  right  hand  side 
of  the  network  (13,15,16,  and  17).  This  seems  reasonable  because  there  were  two 
satellite  links  on  the  left  hand  side  of  the  network  and  one  on  the  right  hand 
side.  The  mutual  system  tends  to  spread  the  variation  over  the  entire  network 
with  smaller  amplitude  at  the  individual  nodes.  It  is  believed  that  this  is  part 
of  the  reason  for  the  observed  performance.  The  other  reason  seems  to  be  in  the 
loop  parameters.  For  all  single  ended  runs  the  mutual  system  had  smaller  phase 
variance  than  the  directed  system.  However,  for  double  ended  runs  the  directed 
system  had  smaller  phase  variance  than  the  mutual  system. 

The  comparisons  5/2,  5/13,  6/2,  and  6/13  show  very  little  difference 
between  the  mutual  single  ended  system  with  a  master  and  the  single  ended  directed 
control  system  with  and  without  self-organizing.  Runs  9/8,  9/14,  10/8,  10/14, 
10/11,  10/15  and  10/16*  show  the  directed  control  system  with  double  ended  and 
other  features  doing  considerably  better  than  the  mutual  control  system  with 
double  ended  and  other  features. 


♦With  measurement  jitter. 


Table  3. 3. 1.4  Comparisons  Between  Mutual  and  Directed  Control  Runs 


Feature  Ratio  Ratio  Ratio  Ratio 

Conbination  of  of  of  of 

"•tlo  <lt*Afl>  <o}&±>  <luA  <t>i> 


Single  Ended 
Mutual  Without 
Master 


MC+EW 

3/2  BC^~ 

3.27 

l.$7 

1.16 

0.83 

,,,,  MC+EW 

3/13  BC+5B 

2.90 

1.65 

1.13 

0.80 

, MC+UEW 

4/2  BCT2— 

3.20 

2.14 

0.82 

0.78 

MC+UEW 

4/13  BC+51T 

2.84 

2.11 

0.80 

0.76 

MC+EW+DOS 

7/2  un - 

5.377 

1.6% 

1.091 

0.808 

MC+EW+DOS 

7/13  BC+50 - 

4.775 

1.678 

1.066 

0.781 

Single  Ended 
Mutual  With 
Master 


c ,,  MC+M+EW 
5/2  - 

MC+M+EW 

5/13  DC+50 

* MC+M+UEW 

6/2  ucr? — 


MC+M+UEW 

6/13  TSC>50 — 

Double  Ended 
MC+DE+EW 

9/8  vcm — 

MC+DE+EW 
9/14  K+UlT+50 

MC+M+DE+UEW+DOS 
10/8  IJC+BE - 

.am.  MC+M+DE+UEW+DOS 

10/14  UC+BF+50 - 

.A...  MC+M+DE+UEW+DOS 

10/11  DC+PE'+rCEWC — 

MC+M+DE+UEW+DOS 

10/15  DC+PE+ICEWC+SO 

<a,i»  MC+M+DE+UEW+DOS 
10/16  nr+DF+irrMir^Br 


1.13 

0.87 

1.003 

0.865 

0.40 

0.89 

0.354 

0.882 

46.019 

1.134 

70.815 

1.209 

5.098 

0.885 

7.845 

0.942 

4.146 

0.914 

9.954 

1.096 

9.92 

1.02 

1.26 

0.81 

1.228 

0.788 

0.54 

0.76 

0.531 

0.736 

3.729 

1.456 

3.804 

2.047 

2.855 

1.142 

2.912 

1.606 

2.696 

1.484 

3.059 

1.665 

3.06 

2.12 

•With  Measurement  Jitter 
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3 . 3 . 1 . 5  Conclusions  for  Maximize  Frequency  Accuracy 

The  ratios  of  the  above  comparisons  were  expressed  In  decibels  with 
appropriate  correction  for  signal  and  power  terms.  The  following  criteria  were 
then  applied: 


2/8 

8/14 

8/11 

11*/12* 


15*/16* 


2/16* 

3/4 


0  dB  <  |  x  |  <#  0.83  dB 
0.83  dB  <  Ixl  <  3.52  dB 
3.52  dB  Slxl  <  6.02  dB 
6.02  dB  <  |x| 


Insignificant 

Slight 

Moderate 

Considerable 


Conclusions  for  Comparisons  Within  Directed  Control 
Double  ended  -  considerable  improvement. 

Automatic  reorganization  -  insignificant  to  slight  improvement. 
Independence  of  clock  error  measurement  and  correction  -  mixed  results  • 
means  degraded  slightly,  variances  improved  slightly. 

Phase  reference  combining  -  insignificant  to  moderate  improvement.  The 
improvement  is  attributed  to  fact  that  fewer  nodes  completely  lost 
reference  under  12* 

Phase  reference  combining  -  insignificant  to  slight  improvement  with 
automatic  reorganization.  Not  as  great  as  11*/12*  because  automatic 
reorganization  shortened  period  during  which  references  were  completely 
lost. 

All  Features  -  considerable  improvement  in  all  parameters. 

Conclusions  for  Comparisons  Within  Mutual  Control 

Unequal  weighting  -  mixed  results.  Insignificant  to  slight  improvement 
in  all  parameters  except  frequency  variance,  slight  degradation  in 
frequency  variance. 
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3/5  Master  in  system  -  mixed  results.  Insignificant  to  considerable 

improvement  in  all  parameters  except  phase  mean.  Insignificant 
degradation  in  phase  mean. 

3/6  Master  and  unequal  weighting  -  insignificant  to  considerable 

improvement  in  all  parameters. 

3/7  Dropout  smoothing  -  mixed  results.  Insi gnif icant  to  moderate 

degradation  in  frequency  errors,  insignificant  improvement  in  phase 
errors. 

3/9  Double  ended  -  mixed  results.  Moderate  degradation  in  frequency  mean 

error,  moderate  to  considerable  improvement  in  other  parameters. 

3/10  All  features  -  moderate  improvement  in  phase  mean.  Considerable 

improvement  in  all  other  parameters. 

Conclusions  For  Comparisons  Between  Mutual  And  Directed 

3/2  Basic  systems  -  mixed  results.  Directed  gives  slight  to  considerable 

improvement  in  frequency  variance  and  mean  and  phase  mean  with 
insignificant  degradation  in  phase  variance. 

3/13  Basic  mutual  versus  directed  with  self-organizing  -  mixed  results. 

Directed  gives  slight  to  considerable  improvement  in  frequency  mean 
and  variance  and  phase  mean  with  slight  degradation  in  phase  variance. 

4/2,  4/13  Mutual  with  unequal  weighting  versus  basic  directed  and  directed 
with  sel f -organ i zat i ng  -  mixed  results.  Directed  gives  slight  to 
considerable  improvement  in  frequency  parameters  but  slight 
degradation  in  phase  parameters. 

7/2,  7/13  Mutual  with  dropout  smoothing  versus  basic  directed  and  directed  with 
self-organizi ng  -  directed  gives  insignificant  to  considerable 
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improvement  in  all  parameters  except  phase  variance  in  which  it  gives 
slight  degradation. 

5/2,  5/13  Mutual  with  master  versus  basic  directed  and  directed  with  self¬ 
organizing  -  mixed  results.  Directed  gave  insignificant  to  slight 
improvement  in  means  but  mutual  gave  insignificant  to  slight 
improvement  in  variances. 

6/2,  6/13  Mutual  with  master  and  unequal  weighting  versus  basic  directed  and 
directed  with  self  organizing  -  mutual  gave  insignificant  to 
considerable  improvement  in  all  parameters. 

9/8,  9/14  Mutual  with  double  ended  versus  directed  with  double  ended  and  directed 
with  double  ended  and  self-organizing.  Directed  gave  insignificant  to 
considerable  improvement  in  all  parameters. 

10/8,  Mutual  with  all  features  versus  directed  with  double  ended  and 

10/14 

directed  with  double  ended  and  self  organizing.  Mixed  results  - 
Directed  gave  insignificant  to  considerable  improvement  in  all 
parameters  except  frequency  variance.  Mutual  gave  insignificant 
improvement  in  frequency  variance. 

10/11  Mutual  with  all  features  versus  directed  with  all  features  except  self 
organizing  and  phase  reference  combining  -  mixed  results.  Directed 
gave  slight  to  considerable  improvement  in  all  parameters  except 
frequency  variance.  Mutual  gave  insignificant  improvement  in  frequency 
vari ance. 

10/15  Mutual  with  all  features  versus  directed  with  all  features  except  phase 
reference  combining.  Directed  gave  insignificant  to  considerable 
improvement  in  all  parameters. 
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10/16*  Mutual  with  all  features  versus  directed  with  all  features  -  Directed 

gave  insignificant  to  considerable  improvement  in  all  parameters. 

Summary  Conclusions  on  Maximize  Frequency  Accuracy 

•  Most  striking  Individual  feature  under  either  mutual  or  directed 
was  double-ended  reference  links. 

•  Directed  control  features  gave  fairly  crisp  (usually  not  mixed) 
improvements  as  individual  features  were  added  with  considerable 
improvement  in  frequency  and  phase  error  means  and  variances  when 
all  features  were  included. 

•  Mutual  control  features  did  not  give  as  crisp  improvements  (often 
mixed)  as  additional  features  did  under  directed,  however,  all 
features  taken  together  gave  considerable  improvements  in  frequency 
and  phase  er-or  means  and  variances. 

•  For  single-ended  runs  there  was  very  little  overall  difference 
between  mutual  and  directed  regimes  but  for  double-ended  runs  the 
directed  system  had  a  definite  advantage  over  the  mutual  system. 

The  reasons  for  this  are  believed  to  be  containment  of  disturbances 
in  the  directed  system  and  the  fact  that  the  Type  1  loop  of  the 
mutual  system  tracks  a  constant  frequency  offset  with  a  nonzero 
phase  error.  It  tracks  a  linearly  drifting  clock  with  constant 
frequency  error  and  constantly  increasing  phase  error. 


*  With  Jitter 
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Higher  Level  Clocks  not  Disturbed  by  Perturbations  at 
Lower  T'evel s  of  the  Network 


36  58 A 


The  low  level  simulation  scenario  runs  were  performed  to  help  evaluate 
this  desirable  characteristic.  The  stress  events  used  in  the  low  level  simulation 
scenario  consisted  of  the  following: 

•  Normal  link  variations  on  link  6-5  from  T  =  0  to  T  =  100,000  sec 

•  Failure  of  Link  6-5  at  100,000  sec 

•  Failure  of  Node  11  at  150,000  sec 

•  Failure  of  Node  3  at  200,000  sec 

•  Step  change  of  +1  x  10'^  in  Node  5 ’ s  natural  clock  frequency  at 
250,000  sec 

The  object  of  this  set  of  simulation  runs  was  to  observe  any  upward 
propagation  in  the  network  as  a  function  of  nodes  from  the  point  of  the 
disturbance  and  as  a  function  of  feature  combinations.  From  the  definition  of 
directed  control  it  may  be  concluded  that  no  such  upward  propagation  can  occur  in 
a  directed  control  system.  The  results  of  the  simulation  verifies  this  logical 
conclusion.  Thus  we  are  left  with  the  evaluation  of  this  characteristic  under  the 
mutual  control  regime. 

Figures  C3LP&F  show  the  basic  mutual  control  run  without  any  of  the 
additional  features.  The  network  disturbance  due  to  the  normal  link  variation  is 
clearly  evident  on  the  frequency  plot.  The  four  events  are  also  clearly  evident 
on  this  plot.  The  frequency  plot  shows  a  gradual  attenuation  and  phase  shift  as 
distance  in  nodes  increases  from  the  source  of  each  disturbance.  Figures  C10LP&F 
show  the  mutual  control  run  with  all  applicable  features. 

3. 3. 2.1  Normal  Link  Variation 

Table  3.3.2. 1  shows  the  propagation  of  the  disturbance  caused  by  the 
normal  link  variation  on  Link  6-5.  All  frequency  errors  were  measured  relative  to 


Table  3. 3. 2.1.  Propagation  of  Disturbance  Upward  Due  to  Event 
Event  =  Normal  Link  Variation  on  Link  6-5 
Amplitude  of  Disturbance  at  Node 
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absolute  frequency  while  all  phase  error  signals  were  measured  relative  to  the 
clock  at  Node  1.  Looking  at  the  phase  error  data  we  see  that  in  going  from  the 
source  of  the  disturbance  toward  Node  1  there  is  attenuation  in  phase  errors,  but 
in  going  from  Node  1  in  the  direction  away  from  the  disturbance  the  errors  tend  to 
increase  again.  This  is  consistent  with  taking  Node  1  as  the  phase  reference. 

The  frequency  error  data  shows  a  gradual  attenuation  as  distance  from  the  source 
of  the  disturbance  increases.  Run  4L  compared  with  Run  3L  shows  the  effect  of 
unequal  weighting.  Run  5L  compared  with  3L  shows  the  effect  of  making  Node  1  the 
master.  Run  6L  shows  the  effect  of  a  master  and  unequal  weighting.  Runs  9L  and 
10L  show  the  effect  of  the  double-ended  feature.  As  indicated  the  double-ended 
feature  reduced  the  propagation  of  the  link  variation  below  the  threshold  of 
measurement.  This  is  the  most  important  information  which  can  be  gained  from 
Table  3. 3. 2.1. 

3. 3. 2. 2  Failure  of  Link  6-5 

Table  3. 3. 2. 2  shows  the  propagation  of  the  disturbance  due  to  the 
failure  of  Link  6-5  at  100,000  sec.  As  seen  from  this  table  there  is  an  overall 
trend  for  a  gradual  decrease  in  amplitude  of  the  disturbance  as  the  distance  from 
the  disturbance  increases  even  though  this  trend  is  by  no  means  monotonic.  The 
most  important  feature  in  decreasing  the  effect  of  this  disturbance  is 
double-ended  reference  links.  This  shows  up  in  Run  9L  compared  with  all  the  runs 
above  it.  The  addition  of  all  the  features  in  Run  10L  provides  some  further 
improvement  over  that  of  Run  9L. 

3. 3. 2. 3  Failure  at  Node  11 

Table  3. 3. 2. 3  shows  the  propagation  of  the  disturbance  due  to  the 
failure  of  Node  11  at  150,000  seconds.  Since  Node  11  was  at  the  very  edge  of  the 
network  and  was  only  directly  connected  to  Node  5  it  made  only  a  minor  impact  on 
the  remai nder  of  the  network.  There  was  a  gradual  attenuation  as  distance  from 
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Table  3. 3. 2. 2.  Propagation  of  Disturbance  Upward  Due  to  Event 
Event  *  Link  6-5  Falls  at  100,000  Seconds 
Amplitude  of  Disturbance  at  Node 
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Node  11  increased.  What  was  at  first  surprising  about  this  disturbance  was  the 
fact  that  the  addition  of  a  master  to  the  network  (Run  5L)  seemed  to  amplify  the 
magnitude  of  the  disturbance.  However  on  second  thought  this  is  not  surprising 
because  without  a  master  the  reference  node  tends  to  move  along  with  the  other 
nodes  for  a  reduction  in  the  amount  of  measured  disturbance.  In  Run  71  the 
dropout  smoothing  reduced  the  effect  of  this  disturbance  below  the  threshold  of 
measurement.  Thus  dropout  smoothing  seemed  to  be  the  most  important  feature 
against  this  particular  disturbance. 

3. 3. 2. 4  Failure  of  Node  3 

Table  3. 3. 2. 4  shows  the  propagation  of  the  disturbance  due  to  the 
failure  of  Node  3  at  200,000  seconds.  The  effects  of  this  failure  were  quite 
similar  to  that  of  the  failure  of  Node  11  at  150,000  seconds  but  the  amplitude  was 
more  pronounced  because  of  Node  3's  higher  position  in  the  network  and  consequent 
greater  connectivity  to  the  remainder  of  the  network.  However,  in  this  case  the 
double  ended  feature  (Run  9L)  provided  greater  attenuation  of  the  disturbance  than 
did  the  dropout  smoothing  (Run  7L). 

3. 3. 2. 5  Clock  Error  at  Node  5 

Table  3. 3. 2. 5  shows  the  propagation  of  the  disturbance  due  to  Node  5's 
clock  making  a  proportional  step  change  of  +1  x  10'9  in  natural  frequency  at 
250,000  seconds.  There  was  some  attenuation  as  a  function  of  distance  from  the 
source  of  the  disturbance  but  this  was  not  monotonic  due  to  the  reference  point  of 
the  network  and  the  uneven  connectivity  of  the  network.  Letting  Node  1  be  the 
master  of  the  network  caused  an  apparent  increase  in  the  amplitude  of  the 
disturbance  but  this  is  merely  a  measurement  illusion  as  explained  in  previous 
discussion.  The  single  most  effective  feature  for  combating  this  disturbance  was 
the  unequal  weighting.  However,  this  is  due  to  the  relatively  low  position  in  the 
network  at  which  Node  5  was  located.  Close  scrutiny  of  Table  3. 3. 2. 5  indicates 


Directed  control  Runs  (21,  81,  and  111  through  161)  had  no  measurable  amount  of  propagation  in  the  network. 
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that  no  combination  of  the  features  was  particularly  effective  in  reducing  the 
propagation  of  this  disturbance. 

3. 3. 2. 6  Conclusions  About  Upward  Propagation  of  Disturbance 

•  Under  Directed  Control 

No  upward  propagation  of  disturbance  with  directed  control 

•  Under  Mutual  Control 

Overall 

A  general  trend  for  amplitude  attenuation  as  distance  from 
source  of  disturbance  increases  but  amplitude  does  not 
monotonically  decrease  with  distance  due  to  connectivity  of 
network  and  location  of  reference  in  network. 

Additional  features  provide  some  further  attenuation  in 
amplitude  of  disturbance.  The  most  important  additional 
feature  is  double-ended  reference  links. 

Normal  Link  Variation 

Double-ended  reduced  the  effect  of  this  disturbance  below  the 
threshold  of  measurement. 

Failure  of  Link  6-5  at  100,000  Seconds 
Double-ended  links  most  important  feature  in  reducing  the 
propagation  of  this  disturbance.  All  additional  features 
provide  a  factor  of  at  least  100  in  reduction  of  frequency 
disturbance  and  between  5  and  50  reduction  in  phase  disturbance. 
Failure  at  Nodes  11  and  3  at  150,000  and  200,000  Seconds 
An  apparent  amplification  of  disturbance  due  to  making  Node  1 
the  master  of  the  network  but  this  is  to  be  expected  because 
when  Node  1  is  not  the  master  it  moves  along  with  the  rest  of 
the  network  and  the  measured  errors  are  smaller  than  they  are 
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when  Node  1  is  stationary.  Dropout  smoothing  was  most 
effective  feature  against  Node  11  failure.  Double-ended 
provided  most  attenuation  for  Node  3  failure  while  dropout 
smoothing  provided  second  most  attenuation. 

Step  Change  in  Node  5's  Clock  of  1  x  10-9  at  250,000  Seconds 

No  combination  of  features  under  mutual  control  was 
particularly  effective  in  reducing  the  propagation  of  this 
disturbance. 

3.3.3  Clock  Errors  Do  Not  Harmfully  Propagate  to  Other  Nodes 

Independent  clock  error  events  were  included  in  each  of  the  simulation 
scenarios.  One  such  event  was  chosen  from  each  scenario  in  order  to  help  evaluate 
this  characteristics.  These  events  were  as  follows: 

•  Low  Level  -  Node  5's  natural  clock  frequency  makes  step  change 

g 

of  +1  x  10"  proportional  parts  at  250,000  s. 

•  High  Level  -  Node  2's  natural  clock  frequency  makes  step  change 

of  +1  x  10"*°  proportional  parts  at  50,000  s. 

•  General  -  Node  6's  natural  clock  frequency  makes  ramp 

increase  of  +1.16  x  10“^  proportional  parts  per 
second  during  the  interval  50,000  -  75,000  s. 

The  method  used  to  evaluate  this  characteristic  was  to  measure  the  propagation  of 
disturbances  as  a  function  of  distance  in  nodes  from  the  source  of  the  disturbance 

and  then  determine  whether  or  not  the  disturbance  is  harmful  by  the  application  of 

the  definition  of  harmful  versus  nonharmful  transient  presented  in  Appendix  0. 
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3. 3. 3.1  Clock  Disturbance  in  Low  Level  Stress  Scenario 
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Table  3.3.3. 1  shows  the  propagation  of  a  clock  disturbance  in  the  low 
level  simulation.  There  is  considerable  propagation  of  the  disturbance  in  the 
mutual  Control  Runs  (3L,  4L,  5L,  6L,  7L,  9L,  10L)  but  very  little  in  the  directed 
control  (2L,  8L,  and  1 1L  through  16L).  The  reason  for  the  limited  propagation  in 
the  directed  control  runs  is  the  fact  that  there  was  only  one  other  monitored  node 
to  which  the  disturbance  could  propagate.  This  was  Node  12.  The  independence  of 
clock  error  measurement  and  correction  feature  in  Runs  11L,  12L,  15L,  and  16L 
completely  removed  this  propagation.  It  is  interesting  to  note  that  the  phase 
errors  at  the  source  of  the  disturbance  over  all  runs  ranged  between  2.1  and 
4.0  (is.  Phase  errors  at  other  nodes  to  which  the  disturbance  propagated  were  all 
less  than  or  equal  to  these  maximums.  It  is  also  noted  that  in  those  directed 
control  runs  for  which  propagation  of  the  disturbance  occurred  the  amplitudes  of 
the  propagated  disturbance  (both  phase  and  frequency)  seemed  to  be  a  function  only 
of  the  loop  parameters.  On  the  other  hand,  in  the  mutual  system  the  amplitudes  of 
the  propagated  errors  were  functions  of  loop  parameters,  network  connectivity,  and 
reference  weighting. 

For  the  directed  control  runs  the  amplitude  of  the  propagated  frequency 
disturbance  one  node  away  was  approximately  0.37  times  that  at  the  source,  while 
the  propagated  phase  error  was  approximately  the  same  as  that  at  the  source.  For 
the  mutual  control  runs  the  amplitude  of  the  propagated  frequency  disturbance 
ranged  from  approximately  one-half  that  of  the  source  one  node  away  to  an 
approximate  average  of  0.036  times  that  at  the  source  three  nodes  away.  The 
amplitude  of  the  propagated  phase  disturbance  had  the  following  averages:  1)  node 
away,  2.03  //s ;  2  nodes  aw  ay,  0.29  ^s;  3  nodes  awa^y,  0.31  /is. 
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3. 3. 3. 2  Clock  Disturbance  in  High  Level  Stress  Scenario 

Table  3. 3. 3.2  shows  the  propagation  of  a  clock  disturbance  in  the  high 
level  scenario.  Due  to  the  size  of  this  disturbance  relative  to  that  of  normal 
link  variations  in  the  single  ended  runs  the  propagated  errors  could  not  be 
measured  with  any  certainty.  Also  in  the  directed  control  runs  there  was  only  one 
node  to  vrfiich  the  disturbance  could  propagate.  This  was  Node  3.  The  independence 
of  clock  error  measurement  and  correction  feature  in  Runs  11H,  12H,  15H  and  16H 
completely  removed  this  propagation.  For  those  directed  control  runs  in  which 
this  error  propagated  to  Node  3  the  phase  error  was  approximately  1.9  ^s  and  the 
frequency  error  was  7.8  x  10"**.  For  the  double  ended  mutual  control  runs  the 
average  frequency  and  phase  propagation  amplitudes  were  as  follows: 

Distance  in  Nodes 

Vi _ i _ 3 

Phase  0.17  0.11  0.11 

Frequency  1.7  x  10"**  1.5  x  10"**  0.94  x  10"** 

3. 3. 3. 3  Clock  Disturbance  in  General  Stress  Scenario 

Table  3. 3. 3. 3  shows  the  propagation  of  a  clock  error  in  the  general 

simulation  runs.  This  error  consisted  of  a  ramp  change  in  Node  6's  natural  clock 

-14 

frequency  of  1.16  x  10  proportional  parts  per  second  over  a  period  of  25,000 
seconds.  This  resulted  in  a  total  natural  frequency  change  of  2.9  x  10"*® 
proportional  parts.  However,  the  local  loop  was  able  to  partially  track  this 
slowly  changing  ramp.  Thus,  all  directed  control  runs  experienced  an  actual 
maximum  output  frequency  change  of  2.45  x  10”**  proportional  parts  and  a  maximum 
phase  error  of  0.57  ^s  at  Node  6.  The  two  double  ended  mutual  control  runs  for 
which  these  changes  could  be  measured  experienced  an  output  frequency  change  of 
3.15  x  10”**  and  1.95  x  10“**  proportional  parts  at  Node  6.  The  average  phase 
error  at  Node  6  for  these  two  mutual  control  runs  was  0.44  ^s.  The  independence 
of  clock  error  measurement  and  correction  feature  in  directed  control  Runs  11G*, 
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Table  3. 3. 3. 2.  Clock  Disturbance  In  High  Level  Simulation 
Event  =  Node  2's  Natural  Clock  Frequency  Up  By  1  x  10"10  at  50,000  Seconds 

Amplitude  of  Disturbance  at  Node 


^  ^  (T  C 

^  -55555-do--- 


5555555?°555555  55555555-555555 


5555555°°555555  5 5 5 5 5 5 5-- 5 5 5 5 5 5 


5555555‘?°555555  5555555 Jd555555 


5555555?°555555  5555555^555555 


5555555dd 555555  5555555-~555555 


5555555?°555555  5555555--55555S 


55555555o555555  555-555--55555S 


*  _  _  W  ro  9)  O' 

-55555-5o55--55 

? 

8  * 

m  a 

^  _  ♦  U  U  ©<_><_> 

:»28  22»T“  ♦TV 
— *  nCdiiwqo* ♦ ♦ ♦ ♦ ♦ ♦ 

I  ♦  ♦  ♦  ♦  ♦  ♦  *UUUUUUU 

3i  8¥*¥¥*Hi'*aaBaHa 
I  £5355£SS§22i?22:2 


CO  -*r«»  00  Ktr  09  00 

r^SSrOoiSr^oMeg  55-‘-:55 


§  2  S£ 

§ii  ii 


a  2  8  2  q  77 

I  UJ  3  ♦  ♦  ♦  UJ  UJ  < 


J  kJ  9  ♦  4-  ♦  uJ  uj  Q  uj  uj  UJ 

^  ♦  IWwioO^QQQ 

^uj9ZZwOOT  4  4  4  4  4  4 

•  ♦  ♦  ♦  ♦  ♦  ♦  *  UUUUUUU 

SS55$5*S§3222SS 


12G#,  15*.  and  16G*  completely  eliminated  the  propagation  of  this  error  to  other 
nodes  of  the  network.  The  other  double  ended  runs  experienced  the  following 
average  propagations  of  this  error  expressed  as  a  fraction  of  the  source. 


\ 

Nodes  Away  From  Source 

1  2 

Directed  Phase 

0.93 

0.63 

Directed  Frequency 

0.99 

0.91 

Mutual  phase 

0.32 

0.21 

Mutual  Frequency 

0.37 

0.52 

Conclusions  for  Clock  Error  Propagation 

•  The  independence  of  clock  error  measurement  and  correction  feature 
in  directed  control  runs  completely  eliminates  the  propagation  of 
these  errors. 

•  In  directed  control  runs  without  ICEM&C  the  propagation  is  confined 
to  the  chain  in  which  the  disturbance  occurs,  propagates  downward 
only,  and  the  amplitude  of  the  propagated  disturbance  is  a  function 
of  the  source  amplitude  and  duration  and  loop  parameters  only. 
Network  frequency  is  not  permanently  affected  unless  source  is 
network  master. 

•  In  mutual  control  runs  these  errors  propagate  throughout  the 
network  and  the  amplitude  of  the  propagated  disturbance  is  a 
complex  function  of  the  amplitude  and  duration  of  the  source,  nodal 
loop  parameters,  reference  weightings  and  network  topology. 

Network  frequency  may  be  permanently  affected  if  source  is 
permanent  and  network  has  no  master. 

•  Frequency  transient  does  not  become  larger  as  it  propagates  through 
the  network  awc^y  from  the  source,  but  rather  experiences 
attenuation. 
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Table  3. 3. 3. 3.  CLock  Disturbance  In  General  Simulation 

Event  ■  Node  6's  Clock  Makes  Ramp  Increase  In  Natural  Frequency 
-14 

of  1.16  x  10  Per  Second  During  Internal  50.000  -  75,000  s 
Amplitude  of  Disturbance  at  Node 
Run 


Number 

51 

II2 

6° 

Phase  ( s) 

8G  DC+DE 

0.53 

0.36 

0.57 

9G  MC+DE+EW 

0.072 

0.014 

0.40 

10G  MC+M+DE+UEW+DOS 

0.21 

0.17 

0.47 

11G  DC+DE+ICEM&C 

UM 

UM 

0.57 

12G*  DC+DE+ICEM&C+PRC 

UM 

UM 

0.57 

14G  DC+DE+SO 

0.53 

0.36 

0.57 

15G  DC+DE+ICEM&C+SO 

UM 

UM 

0.57 

16G*  DC+0  E+ IC  EM&C+PRC+SO 

UM 

UM 

0.57 

Frequency  (Proportional 

Parts) 

8G  DC+DE 

2.43  ) 

c  10-U 

2.28  x  10-H 

2.46 

x  10-11 

96  MC+DE+EW 

3.0  x 

1°“}J 

1.59  x  10-H 

3.15 

x  10-11 

10G  MC+M+DE+UEW+OOS 

8.4  x 

10-12 

1.04  x  10-H 

1.95 

x  10-11 

11G  DC+DE+ICEM&C 

UM 

UM 

2.46 

x  10-11 

12G*  DC+DE+ICEM&C+PRC 

UM 

UM 

2.46 

x  10-11 

14G  DC+DE+SO 

2.4  x 

lo-n 

2.2  x  10-H 

2.46 

x  10-11 

15G  DC+DE+ICEM&C+SO 

UM 

UM 

2.46 

x  10-11 

16G  DC+DE+ ICEM&C+PRC+SO 

UM 

UM 

2.46 

x  10-H 

*With  Jitter 


Propagation  of  disturbance  could  not  be  measured  in  single  ended  Run  (2G  through 
7G  and  13G)  due  to  being  swamped  by  normal  link  variations. 
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The  rise  time  of  the  frequency  transient  is  fastest  at  the  source. 


In  this  regard  a  unit  step  has  fastest  rise  time. 

Whether  or  not  the  disturbance  is  harmful  depends  on  the  amplitude 
and  duration  of  the  source,  how  frequently  it  occurs,  and  whether 
or  not  it  occurs  as  an  isolated  event  or  in  consonance  with  other 
disturbances. 

When  the  definition  of  harmful  transient  (which  is  concerned  with 
the  loss  of  receiver  bit  timing  due  to  not  being  able  to  rapidly 
follow  large  frequency  changes  in  received  signals  or  design 
compromises  in  signal-to-noise  ratio  to  avoid  the  problem)  the 
following  conclusions  can  be  made: 

The  transient  in  the  low  level  simulation  of  +1  x  10"9  could  be 
potentially  harmful  to  all  imnediate  neighbors  of  Node  5  under 
mutual  control  and  to  Nodes  11  and  12  under  directed  control 
without  the  feature  of  independence  of  clock  error  measurement  and 
correction. 

The  transient  in  the  high  level  simulation  of  1  x  10"10  would  be 
unlikely  to  cause  observable  problems  under  either  mutual  or 
directed  control  at  any  node  of  the  network. 

The  ramp  change  in  the  general  simulation  of  1.16  x  10'14 
proportional  parts  per  second  would  have  no  effect  on  slip  rate  or 


3*3.4  lloaes061**  Var1atl0ns  and  Pr°PQuts  Do  not  Harmfully  Propagate  to  Other 

The  high  level  simulation  scenario  runs  were  used  to  help  evaluate  this 
desirable  characteristic.  This  set  of  runs  contained  an  Initial  transient,  one 
satellite  link  variation,  one  Independent  clock  error  transient,  one  link  dropout 
and  failure  of  the  master  or  reference  node. 

3. 3.4.1  Satellite  Link  Variation 

The  satellite  link  variation  was  applied  on  Link  1-13  from  the 
beginning  of  the  simulation  until  failure  of  this  link  at  150,000  seconds.  This 
variation  was  modeled  as  a  sinusoid  of  period  one  day.  The  equation  for  this 
variation  was 


AL  *  2.52  x  10-5  x  LQ  x  sjn  (wdt) 

with  L0  -  71,280  km 

and  cjd  .  7.2685  x  10‘5  rad/s- 

With  a  nominal  propagation  speed  of  3  x  105  km/sec  this  resulted  in  a 
peak  phase  variation  of 


ATmax 


2.52  x  10'5  x  71,280  km 


3  x  103 


km/s 


■  5.99  x  10  seconds 


The  resulting  peak  frequency  Doppler  was 


*max  *  2.52  x  10 


~5  x  71,280  x  7.2685  x  10 
3  x  10* 


-5 


4.35  x  10-10  proportional  parts 

Table  3.3.4. 1  shows  the  propagation  of  this  disturbance  to  the  nodes  of 
the  network  In  the  form  of  peak  phase  and  frequency  variations.  All  double  ended 
runs  completely  eliminated  the  propagation  of  this  disturbance.  Thus  we  are  left 
with  evaluation  of  this  disturbance  for  the  single  ended  runs.  From  this  table  we 


3671A 


3-76 


1 

I 

I 

I 

I 

I 

I 

I 

[ 

I 

I 

I 

t 


3658A 

see  that  under  directed  control  (2H  and  13H)  the  disturbance  propagated  only  down 
the  chain  in  which  the  source  was  located.  There  was  attenuation  at  Node  13  since 
this  cesium  clock  node  had  loop  parameters  which  provides  an  attenuation  of  0.5343 
to  daily  frequency  variations.  There  was  no  further  frequency  attenuation  at 
Nodes  16  and  17  because  these  crystal  clock  nodes  had  a  gain  of  1  for  daily 
frequency  variations. 

In  the  single  end  mutual  control  runs  this  disturbance  propagated  to 
all  nodes.  The  nodal  loop  gain  to  daily  frequency  variations  was  equal  to  1.0  for 
all  mutual  nodes  but  none  of  the  nodes  experienced  the  maximum  variation  of  4.35  x 
10"*°  of  the  source  because  of  the  anchoring  effect  of  neighboring  nodes.  The 
average  peak  disturbances  related  to  distance  in  nodes  from  the  source  of  the 
disturbance  is  shown  below. 


Distance  from 

Source  in  Nodes 

Mutual 

1 

2 

3  4 

Average  Peak  Phase 

2.50 

1.18 

0.80  0.47 

Average  Peak  Frequency 

2.5  x  10‘10 

1.86  x  10-10 

1.72  x  10“10  1.48  x  10 

Di rected 

Average  Peak  Phase 

3.51 

3.51 

3.35 

Average  Peak  Frequency 

2.4  x  10"10 

2.4  x  10'10 

2.4  x  10"10 

3. 3.4.2  Link  Dropout 

Link  1-13  failed  at  150,000  seconds.  Table  3. 3. 4. 2  shows  the 
propagation  of  this  disturbance.  In  the  directed  control  runs  this  disturbance 
propagated  at  most  down  the  branch  in  which  the  disturbance  occurred.  However,  no 
propagation  of  this  disturbance  was  experienced  in  Runs  12H*  and  16H*.  In  runs 
11H  and  15H  drop-in-smoothing  was  applied  only  to  one  node  below  the  point  of  the 
disturbance,  in  this  case  Node  13.  Nodes  16  and  17,  lower  in  the  chain  than  Node 
13,  did  not  have  smoothing  applied  because  even  though  they  lost  the  network 


- 


3-79 


A 


3658A 

master  they  did  not  lose  their  immediate  reference.  Nodes  13  and  16  respectively, 

when  Link  1-13  failed.  Node  13  initially  had  a  frequency  offset  of  +1  part  in 

10**  and  even  though  it  had  acquired  the  master  the  integrator  in  its  control 

loop  had  not  had  time  at  150,000  seconds  to  come  to  the  proper  value  to  hold  the 

offset.  Thus,  when  Link  1-13  failed  at  150,000  seconds  Node  13  reverted  back  to  a 

frequency  error  of  -6.8  x  10  with  respect  to  Node  1.  Nodes  16  and  17 

partially  followed  Node  13  to  this  new  value  during  the  coasting  interval  and 

thereby  built  up  some  phase  error  relative  to  Node  1  before  Node  13  changed 

references  to  Node  2  at  157,200  seconds.  Node  13  then  had  smoothing  applied  to 

reacquire  the  ultimate  master  via  Node  2  but  Nodes  16  and  17,  not  having  smoothing 

applied,  experienced  frequency  spikes  of  1.5  x  10”**  and  2.1  x  10'** 

respectively  upon  reacquiring  the  ultimate  master  via  Nodes  13  and  2.  In  Run  15H 

the  sequence  of  events  were  identical  to  those  in  Run  11H  except  the  coasting 

interval  was  only  300  seconds  as  opposed  to  7,200  seconds  in  Run  11H.  Therefore 

in  Run  15H  Nodes  16  and  17  did  not  follow  Node  13  when  it  reverted  back  to  an 
12 

error  of  -6.8  x  10"  c  proportional  parts.  Thus  a  much  smaller  phase  error  was 
built  up  with  respect  to  Node  1  at  Nodes  16  and  17.  This  accounts  for  the  much 
smaller  frequency  spikes  at  these  nodes  in  Run  15H.  The  above  action  was  observed 
with  the  aid  of  the  line  printer  plots  since  the  resolution  of  the  CALCOMP  plots 
was  too  coarse  to  see  any  of  these  spikes. 

Under  mutual  control,  the  disturbance  due  to  this  link  dropout 
propagated  to  almost  all  nodes  of  the  network.  The  most  effective  feature  for 
reducing  the  peak  amplitudes  of  the  disturbance  under  mutual  control  was  the 
double  ended  feature.  The  average  peak  amplitude  over  all  combinations  of 
features  is  shown  below. 


Superscripts  indicate  distance  in  nodes  from  source  of  disturbance 


Table  3. 3. 4. 2.  Propagation  of  Disturbance  Due  to  Link  Failure 
Peak  Amplitude  of  Disturbance  at  Node  (Continued) 
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Distance  from 

Source  in  Nodes 

Mutual 

1 

2 

3 

4 

Average  Peak  Phase 

1.34 

0.48 

0.24 

0.13 

Average  Peak  Frequency 

4.13  x  10"10 

• 

X 

M 

o 

»— • 
o 

0.89  x  10'10 

0.52  x  10 

Di  rected 

Average  Peak  Phase 

0.26 

0.26 

0.20 

- 

Average  Peak  Frequency 

6.1  x  10'11 

1.5  x  10'11 

2.1  x  10’11 

3. 3. 4. 3  Network  Master  Fails 

Failure  of  the  network  master  represented  a  major  disturbance  in  the 
network.  The  propagation  of  this  disturbance  throughout  the  network  is  shown  in 
Table  3. 3. 4. 3.  All  frequency  measurements  were  made  relative  to  absolute  zero 
offset.  All  phase  measurements  were  made  relative  to  Node  1  until  failure  of  Node 
1  and  thereafter  relative  to  Node  2.  From  this  table  it  is  concluded  that 
directed  control  was  the  most  effective  feature  in  reducing  the  amplitude  of  the 
propagated  disturbance.  The  average  peak  errors  over  the  runs  under  mutual 
control  and  directed  control  as  related  to  distance  from  the  source  of  the 
disturbance  is  shown  below. 


Distance  in  Nodes  from  Source 


Mutual 

1 

2 

3 

Average  Peak 

Phase 

1.39 

1.05 

0.84 

Average  Peak 

Frequency 

8.39  x 

10'11 

4.46  x 

io-11 

2.31  x  10 

Di rected 

Average  Peak 

Phase 

0.16 

0.12 

0.26 

Average  Peak 

Frequency 

1.29  x 

10"11 

9.08  x 

10~12 

1.52  x  10 

I 
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3. 3. 4.4 


Conclusions  About  Path  Delay  Variations  or  Dropouts 

Normal  path  delay  variations: 

•  Double-ended  reference  links  completely  removes  the  effects  of 
normal  path  delay  variations. 

•  Path  delay  variations  propagate  down  chain  from  source  in  directed 
control  but  throughout  the  network  with  mutual  control  with  single 
ended  reference  links.  The  variations  are  smaller  with  mutual 


control  at  individual  nodes  because  neighbor  nodes  tend  to  act  as 
anchors. 


The  applied  variation  was  1/200  of  the  maximum  that  can  be  expected 
from  a  military  satellite  (3°  inclination,  eccentricity  of  0.01 
with  earth  station  at  80°  latitude).  This  resulted  in  a  source 
disturbance  with  peaks  of: 

Phase  -  6  fis 

Frequency  -  4.35  x  10'^  p-p 

Application  of  the  definition  of  harmful  transient  to  the  peak 
errors  indicates  that  they  are  unlikely  to  cause  observable 
problems.  Even  if  the  amplitude  of  Doppler  variations  of  signals 
which  pass  through  satellite  terminals  to  other  nodes  of  the 
terrestrial  network  should  increase  by  small  amounts.  It  is  not 
expected  that  they  would  cause  a  problem  for  receiver  bit  recovery 
loops  because  of  their  low  frequency.  However,  for  large  amounts 
of  increase  (3  10  or  greater)  there  could  be  some  harmful  effects 
due  to  peak  amplitude  and  rate  of  change.  For  example,  at  the 
1/200  value  the  maximum  rate  of  change  is  3.14  x  10~*4  parts  per 


second.  This  may  be  compared  with  the  clock  ramp  change  of  1.16  x 
10-*4  parts  per  second  evaluated  in  the  previous  section  which 
caused  peak  frequency  errors  of  3.15  x  10-^  p-p. 
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Table  3. 3. 4. 3.  Propagation  of  Disturbance  Due  to  Failure  of  Network  Master 
Peak  Amplitude  of  Disturbance  at  Node  (Continued) 
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Link  Dropout  1-13 

•  For  this  particular  dropout  the  combination  of  features  directed 
control,  double  ended,  independence  of  clock  error  measurement  and 
correction  and  phase  reference  combining  completely  eliminated  the 
effects  of  this  disturbance.  This  is  primarily  attributed  to  the 
phase  reference  combining  feature  which  ensured  that  no  node  in  the 
chain  below  the  disturbance  was  completely  without  a  reference. 

All  other  combinations  of  features  showed  some  propagation  of  this 
d i sturbance. 

•  The  double  ended  feature  was  the  single  most  important  feature  in 
reducing  the  effects  of  this  disturbance.  It  provided  a  reduction 
of  almost  10  under  the  mutual  regime  and  greater  than  10  under  the 
directed  regime. 

•  The  dropout  smoothing  feature  provided  an  attenuation  of  up  to  four 
at  some  nodes  under  the  mutual  control  regime. 

•  Application  of  the  definition  of  harmful  transient  shows  that  the 
only  combinations  of  features  for  which  potentially  harmful  effects 
would  occur  was  mutual  control  with  unequal  weighting  of  reference 
links.  In  this  run  a  frequency  transient  of  1.1  x  10”^  p-p 
occurred  at  Node  13. 

Failure  of  Node  1 

•  No  combination  of  features  completely  eliminated  the  propagation  of 
this  disturbance  throughout  the  network. 

•  The  directed  control  runs  had  considerably  smaller  phase  errors 
than  the  mutual  control  runs. 

•  The  largest  frequency  transient  as  a  result  of  this  disturbance  was 
4.81  x  10  p-p  in  Run  5H  (mutual  control  with  master  at  Node 

5).  Application  of  the  definition  of  harmful  transient  to  this 


peak  disturbance  indicates  that  such  a  disturbance  would  be  unlikely  to 
cause  observable  problems  with  receiver  bit  recovery  loops  or  result  in 
any  sacrifice  in  SNR  to  accommodate  the  tracking  of  such  changes. 

3. 3. 4. 5  Better  Model  for  Tropo  Links 

Comparison  of  the  model  for  tropo  links  of  Table  3. 2. 3. 9  with  recently 
available  measured  data  in  reference  223  Indicates  that  the  model  for  tropo  links 
used  in  the  simulation  was  not  a  good  model.  Analysis  of  the  data  of  in  reference 
221  Indicates  that  the  equation: 

AL  =  1  x  10-4  Lq  sin  (7>3  x  10-5  xt)  +  (3. 3. 4. 5-1) 

2  x  10-5  Lq  sin  (8>7  x  10-4  t) 

would  be  a  much  better  model.  If  this  model  were  used  then  peak  phase  error  and 
peak  frequency  Doppler  resulting  from  tropo  link  variations  would  be: 

— f-Pea-k  *  1  ---x-c—  31  0.4  ns/km  (3. 3. 4. 5-2) 

Lo  3  x  10s 


Af  peak 

nrr 

n  o 


24.7  x  10' 


3  x  103 


8.23  10'14  p-p/km 


(3. 3. 4. 5-3) 


Four  tropo  links  were  used  in  the  simulation  ranging  from  1000  km  to 
1500  km  in  length.  The  longest  tropo  link  would  then  give  a  peak  phase  variation 
of  0.6  ps  and  a  peak  frequency  Doppler  of  1.23  x  l.O-^  p-p.  These  numbers 
represent  approximately  10  percent  of  peak  phase  error  seen  on  satellite  links  and 
28  percent  of  peak  frequency  Doppler  seen  on  satellite  links.  Thus,  without 
consideration  of  the  percentage  of  links  affected  the  effects  of  using  the  correct 
model  for  tropo  links  is  expected  to  be  as  follows: 

•  Double-ended  runs  would  not  be  affected 

•  Single-ended  runs  would  have  shown  peak  phase  errors  at  some  nodes 

no  more  than  10  percent  worse  than  were  actually  seen  with 

incorrect  model,  and  peak  frequency  errors  on  some  links  no  more 
than  28  percent  worse  than  those  actually  seen. 
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Slip  Free 

The  major  purpose  of  the  disciplining  features  is  to  provide  the 
digital  DCS  communications  network  a  normal  mode  of  operation  that  is  free  of  bit 
slips.  The  simulations  which  were  performed  provide  some  indication  of  the  size 
of  data  buffers  needed  to  accomplish  slip  free  operation.  The  runs  of  the  general 
simulation  scenario  were  used  to  help  evaluate  this  desirable  characteristic.  The 
following  information  was  obtained  from  these  simulation  runs. 

•  Peak  phase  errors  due  to  initial  transients 

•  Peak  phase  errors  due  to  normal  link  variations 

•  Peak  phase  errors  due  to  link  and  nodal  failures 

•  Phase  errors  due  to  drifting  clocks 

3. 3. 5.1  Peak  Phase  Errors  Due  to  Initial  Transients 

An  initial  transient  was  applied  with  all  runs  of  the  general 
simulation  scenario.  This  initial  transient  consisted  of  an  initial  frequency 
offset  at  selected  nodes  of  the  network.  These  offsets  are  listed  in  Table 

o 

3. 2. 3. 6-1.  They  range  from  0  at  Node  1  to  1  x  10  proportional  parts  at  Nodes 
17  and  18.  Table  3. 3. 5.1  shows  peak  phase  errors  at  three  nodes  of  the  network  as 
a  result  of  these  initial  transients.  Other  nodes  of  the  network  experienced 
smaller  initial  transients  than  these  nodes.  Only  double  ended  runs  were  used 
because  these  initial  transients  were  swamped  by  normal  satellite  link  variations 
of  single  ended  runs.  It  is  noted  that  since  the  simulated  system  was  linear, 
proportionally  larger  initial  offsets  would  produce  proportionally  larger  peak 
phase  errors,  but  time  constants  would  not  be  affected. 
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Table  3. 3. 5.1.  Peak  Phase  Errors  Due  to  Initial  Transients 


Run 

Peak  Phase  Error  at  Node  (  ^is) 

Number 

11 

16 

17 

8G 

DC+DE 

1.0 

0.25 

2.03 

9G 

MC+DE+EW 

0.93 

1.52 

2.8 

10G 

MC+M+DE+UEW+DOS 

0.93 

0.58 

1.8 

1 1 G 

0C+DE+ICEM8.C 

1.06 

0.05 

2.2 

12G* 

DC+DE+ICEMfcC+PRC 

1.06 

0.08 

2.15 

14G 

DC+DE+SO 

1.1 

0.28 

2.1 

15G 

DC+DE+ICEM8.C+S0 

1.05 

0.05 

2.12 

16G* 

DC+DE+  ICEM8.C+PRC+S0 

1.05 

0.075 

2.12 

*With 

jitter 

3.3.5. 

2  Peak  Phase  Errors  Due 

to  Normal 

Link  Variations 

Normal  link  variations  were  applied  to  all  links  in  all  general 
simulation  stress  scenario  runs.  However,  the  resulting  satellite  variations 
dominated  because  they  were  at  least  a  couple  of  orders  of  magnitude  larger  than 
variations  on  other  links  in  spite  of  the  fact  that  the  simulation  model  used  only 
l/200th  of  the  actual  maximum  variations  on  satellite  links.  This  was  done 
because  recent  procurements  for  equipment  to  instrument  satellite  ground  terminals 
to  handle  digital  data  have  included  "Satellite  Buffers"  which  will  use  double 
endedness  to  accomodate  the  majority  of  common  mode  transmission  time  delay 
variation  expected  due  to  daily  vertical  drift.  If  these  buffers  and  controls  do 
not  exist  at  satellite  ground  terminals  then  the  200:1  reduction  of  actual  maximum 
variations  on  satellite  links  must  be  reevaluated  requiring  much  larger  buffers  at 
the  ends  of  satellite  links  when  double  endedness  is  used  and  requiring  larger 
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buffers  throughout  the  network  If  double  endedness  is  not  used.  Table  3. 3. 5. 2 
shows  the  resulting  peak  phase  errors  in  single  ended  runs.  The  largest  such 
error  observed  was  13.30  /is  in  run  13  G  at  Node  11.  This  node  was  disciplined 
through  two  tandem  satellite  links. 

Table  3.3. 5.2.  Peak  Phase  Errors  Due  to  Normal  Link  Variations 


Run 

Number 

2 

3 

Peak  Phase 
5  6 

Error 

7 

at  Node 
11 

Us) 

13 

15 

16 

17 

2G  OC-2 

0 

0 

12.08 

6.12 

0 

11.97 

3.30 

0 

3.30 

3.09 

3G  MC+EW 

1.25 

2.14 

0.98 

2.85 

1.87 

1.61 

2.32 

3.74 

2.85 

4.81 

4G  MCHIEW 

1.27 

2.19 

0.72 

2.42 

1.67 

1.55 

0.58 

3.11 

2.93 

3.68 

5G  MC+M+EW 

3.79 

4.64 

5.56 

6.80 

2.16 

5.49 

3.27 

2.09 

2.58 

2.55 

6G  MC+M+UEW 

3.02 

3.98 

5.21 

6.12 

1.67 

5.21 

3.65 

2.50 

3.21 

3.13 

7G  MC+EW+DOS 

1.21 

2.10 

1.05 

2.83 

1.90 

1.70 

1.46 

3.72 

4.16 

4.77 

13G  OC+SO 

0 

0 

13.07 

6.71 

0 

13.30 

3.27 

0 

3.27 

3.50 

3. 3. 5. 3  Peak  Phase  Errors  Due  to  Link  and  Nodal  Failures 

During  the  general  simulation  stress  scenario  runs  Link  6-5  failed  at 
150,000  seconds.  Link  1-2  failed  at  200,000  seconds  and  Node  13  failed  at  250,000 
seconds.  These  link  and  node  failures  were  used  to  get  an  indication  of  the  size 
of  phase  errors  resulting  from  such  stresses.  Table  3. 3. 5. 3  shows  the  peak  errors 
in  two  directed  control  double  ended  runs  (8G  and  11G)  and  one  mutual  control 
double  ended  run  (9G).  These  errors  were  swamped  in  the  single  ended  runs  by  the 
normal  satellite  link  variations.  Other  double  ended  runs  showed  results  similar 
to  those  of  the  table.  The  largest  observed  error  was  2.244  fis  as  a  result  of 
Mode  13 ’ s  failure  in  the  mutual  control  in  9G.  The  largest  observed  error  under 
directed  control  was  0.86  as  a  result  of  Node  13 's  failure  in  Run  UG. 
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Table  3. 3. 5. 3.  Peak  Phase  Errors  Oue  to  Link  and  Nodal  Failures 
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3. 3. 5. 4  Phase  Errors  Oue  to  Drifting  Clocks 
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This  source  of  phase  error  does  not  appear  to  be  a  problem  with  the 


Tjpe  2  loops  used  in  the  directed  control  runs  because  these  effects  are  taken 


care  of  by  the  integrator  of  this  loop.  However,  it  is  a  problem  for  the  Type  1 


phase  locked  loops  used  with  all  mutual  control  runs. 


For  the  Type  1  phase  locked  loop  a  constant  frequency  drift  of  & 


proportional  parts  per  unit  time  will  produce  a  constantly  increasing  phase  error 


of  the  form 


d>e(t) 


(3. 3. 5.4) 


With  ■  ID’10  x  fQ  per  day  or  1.1574  x  10“^  proportional  parts 


second,  {  *  1,  and  <*>n  -  1.52  x  10“J  rad/sec  the  following  phase  errors  will 
accumulate  during  the  indicated  time  period. 


Phase  Error  <6e 


1  day 


0.132  ns 


10  days 


1.32  ns 


100  days 


13.2  ns 


1000  days 


Such  an  accumulation  of  phase  errors  can  be  observed  in  the  mutual  control  runs. 


For  example  in  run  106  the  entire  network  appears  to  accumulate  phase  error 
relative  to  Node  1  of  approximately  0.351  ps  in  300,000  seconds  of  simulated  run 


time.  This  is  an  average  of  0.1011  ps  per  day.  This  accumulation  of  phase  error 


3-93 


3658A 

is  the  result  of  the  drift  of  the  quartz  clocks  of  the  network.  The  magnitude  of 
this  drift  ranged  from  0  at  Nodes  1,  2,  13,  and  14  to  1  part  in  1010  per  day  at 
Nodes  4,  10,  and  17.  The  arithmetic  average  over  the  20  nodes  of  the  network  was 
+1.25  parts  in  1011  per  day. 

3. 3. 5. 5  Peak  Composite  Phase  Errors 

In  order  to  be  safe,  buffering  sufficient  to  handle  a  composite  of 
phase  error  resulting  from  a  sum  of  the  magnitudes  of  the  peak  errors  of  the  four 
sources  considered  above,  must  be  provided.  These  are  listed  below: 

•  Mutual  single  ended  -  Initial  transient  +  normal  link  variations 

+  link  and  node  failure  +  clock  drift 
=  2.8  +  6.8  +  2.244  +  0.132  u s/day 
=  11.844  +  0.132  //s/day 

•  Mutual  double  ended  -  Initial  transient  +  link  and  node  failure  + 

clock  drift 

=  2.8  +  2.244  +  0.132  //s/day 
=  5.044  +  0.132  // s/day 

•  Directed  single  ended  -  Initial  transient  +  normal  link  variations 

+  link  and  node  failure 
*  2.2  +  13.30  +  0.86  =  16.36  /is 

•  Directed  double  ended  -  Initial  transient  +  link  and  node  failure 

=  2.2  +  0.86  =  3.06  /is 

Let  us  now  assume  that  the  normal  operational  procedure  used  with  the  mutual 
system  allows  for  special  provisions  with  crystal  clocks  to  cancel  accumulated 
phase  error  every  100  days.  These  provisions  are  discussed  in  Paragraph  3. 3. 5. 6. 
Since  the  errors  which  were  measured  were  all  relative  to  Node  1  it  is  conceivable 
that  twice  this  amount  of  error  may  result  between  two  communicating  nodes.  The 
errors  may  also  be  plus  or  minus  which  requires  a  further  multiplication  by  two. 
All  this  results  in  multiplication  of  the  values  given  above  by  four.  Thus,  Table 
3.3. 5.5  shows  the  buffer  sizes  to  operate  slip  free  under  all  the  considerations 
of  this  section. 
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Table  3. 3. 5. 5.  Buffer  Sizes  for  Slip  Free  Operation 


Disciplining 

System  Buffer  Sizes 

+  Mutual  Single  Ended  100  /is 

+  Mutual  Double  Ended  73  n s 

Directed  Single  Ended  63  fjs 

Directed  Double  Ended  13  //s 


+  Requires  special  provisions  with  crystal  clocks  to  cancel  accumulated 
phase  error  every  100  days. 

3. 3. 5. 6  Special  Provisions  With  Crystal  Clocks  and  Type  1  Loops  in  Mutual 
System  for  Handling  Buildup  of  Phase  Error 

Three  techniques  have  been  suggested  in  regards  to  this  problem.  They 
are  as  follows: 

1.  Size  buffers  large  enough  to  handle  lifetime  drift  of  20  years 

2.  Mechanically  readjust  center  frequency  of  oscillator  at  periodic 
intervals 

3.  Program  nodal  processor  to  automatically  cancel  phase  error  through 
control  of  nodal  frequency 

3. 3. 5. 6.1  Lifetime  Buffers 

The  rate  at  which  data  buffers  will  fill  with  a  drifting  clock  and 
second  order  Type  1  nodal  loop  is  given  by  Equation  3. 3. 5. 4.  Using  the  parameters 
of  the  simulation  and  a  drift  rate  of  10'10  pp/day  requires  a  buffer  of 
approximately  964  //s  for  a  20-year  interval.  Multiplying  this  by  a  factor  of  four 
to  handle  node  to  node  communications,  and  either  positive  or  negative  drift,  we 
require  a  buffer  of  3857  fis.  Although  buffers  of  similar  size  may  be  required  at 
each  end  of  satellite  links,  it  is  believed  that  the  amount  of  delay  variation 
implied  by  this  large  a  buffer  at  each  crystal  clock  node  of  the  terrestrial 
portion  of  the  DCS  network  would  be  undesirable. 
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3. 3. 5. 6. 2  Mechanical  Readjustment 

If  mechanical  readjustment  of  the  center  frequency  is  performed  every 
100  days,  then  buffers  of  the  size  indicated  in  Table  3. 3. 5. 5  are  sufficient. 
However,  mechanical  readjustment  in  the  DCS  network  is  undesirable  from  an 
operational  manpower  requirement. 

3. 3. 5. 6. 3  Processor  Controlled 

Figure  3.3. 5.6.3  shows  a  single  phase  comparator  Type  1  loop  with  a 
processor  activated  phase  error  controller.  During  the  interval  when  the 
threshold  device  is  open  the  circuit  behaves  as  a  normal  Type  1  loop  with  an 
unconditionally  stable  mutual  network  provided  {>0.707.  With  the  threshold 
device  closed  the  closed  loop  transfer  function  is 

flo(S)  +  Kv  (b  ♦  a)  S  +  Kvab  (3. 3. 5.6. 3-1) 

0i(S)  S3  ♦  aS*  +  Ky(b  +  a)  S  ♦  Kyab 

The  stability  criterion  in  the  mutual  system  requires 

|  <1  (3. 3. 5. 6. 3-2) 

for  all  o  #  o 

After  some  algebraic  manipulation  we  find  that  the  stability  criterion  Is  not  met 
for 

2a2bKy  +  ^2  [2Ky(b  +  a)  -  a2]  -  w4  >  o.  (3. 3. 5. 6. 3-3) 
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Figure  3. 3. 5. 6. 3.  Type  One  Loop  With  Processor  Activated 
Phase  Error  Controller 
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Thus  there  are  nonzero  values  of  cj  for  which  the  system  does  not  meet  the  network 


stability  criterion.  This  is  not  to  say  that  the  network  would  definitely  be 
unstable  with  the  threshold  device  closed,  but  one  would  certainly  not  design  a 
system  until  all  other  factors  which  might  bear  on  this  stability  problem  had  been 
evaluated  and  been  determined  to  be  controllable.  These  factors  are  concerned 
with  link  delays  and  network  topology.  No  general  set  of  conditions  is  known 
which  provides  the  desired  degree  of  confidence  that  the  network  would  remain 
stable  with  such  a  scheme  for  phase  error  control.  Thus  it  is  not  certain  that 
such  a  solution  can  be  realized. 

3.3.6  Survi vabi 1 ity 

This  desirable  characteristic  is  a  measure  of  the  system's  ability  to 
continue  to  operate  under  stressed  conditions.  If  intrinsically  perfect  clocks 
could  be  provided  at  each  node,  survivability  would  be  provided  to  the  utopian 
degree.  However,  perfect  clocks  are  not  available.  Without  perfect  clocks,  it  is 
not  a  foregone  conclusion  that  a  network  of  independent,  imperfect  clocks  will 
provide  a  more  survivable  network  than  a  network  of  disciplined  imperfect  clocks. 
On  the  contrary,  it  seems  reasonable  that,  with  clock  quality  fixed,  some  scheme 
which  is  able  to  make  all  clocks  of  the  network  identically  imperfect  will  improve 
survivability  over  the  former  method.  However,  it  is  not  the  purpose  of  this 
section  to  prove  or  disprove  such  a  conjecture.  The  purpose  of  this  section  is  to 
attempt  to  assess  the  relative  degree  to  which  the  synchronization  subsystem 
features  provide  a  survivable  network.  Certainly  the  disciplining  features  of 
mutual  control  and  directed  control  are  important  with  this  latter  approach  to 
network  survivability. 
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Under  the  disciplined  approaches  two  major  operating  intervals  for 
performance  evaluation  can  be  identified. 

Case  1  -  References  are  available 

Case  2  -  References  are  not  available 

In  Case  1,  conditions  for  mutual  and  directed  control  (without  phase 
reference  combining)  are  considerably  different  because  each  node  in  the  mutual 
system  is  always  using  all  of  its  available  references  whereas  a  node  in  the 
directed  control  system  uses  only  one  of  its  available  references  at  a  given 
time.  If  one  of  the  references  is  lost  some  form  of  transient  is  inevitable  in 
the  mutual  system  unless  the  error  of  the  local  node  with  respect  to  the  lost 
reference  was  exactly  zero  at  the  time  this  reference  was  lost.  In  the  directed 
system,  if  the  local  node  did  not  happen  to  be  using  the  reference  at  the  time 
that  it  was  lost,  no  such  transient  results.  Also,  if  there  is  an  alternate 
reference  and  there  is  no  error  difference  between  the  old  reference  and  the 
alternate  reference,  then  no  transient  results.  Thus,  we  see  that  in  either 
disciplining  regime  those  features  which  tend  to  minimize  instantaneous  frequency 
and  phase  errors  will  tend  to  improve  network  survivability.  Under  the  directed 
regime,  the  quicker  a  new  reference  can  be  selected  once  an  old  reference  has  been 
lost,  the  better  from  a  survivability  standpoint.  Thus,  automatic  reorganization 
should  be  quite  important  to  survivability  of  the  directed  system. 

3. 3.6.1  Case  2  -  References  Are  Not  Available 

In  Case  2,  in  which  the  last  remaining  reference  has  been  temporarily 
lost,  the  mutual  and  directed  regimes  have  similar  problems.  They  both  must  run 
asynchronously  until  a  reference  can  be  restored.  In  this  case  both  must  start 
with  accuracy  of  the  local  clock  immediately  prior  to  beginning  asynchronous 
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operation.  They  must  then  rely  on  the  stability  of  the  local  clock  coupled  with 
any  information  that  the  local  disciplining  technique  can  remember,  and  utilize 
about  the  long  term  history  of  its  error  with  respect  to  its  reference,  in  order 
to  coast  through  the  outage  and  still  remain  closely  enough  in  synchronism  to 
avoid  slips  when  a  reference  is  restored. 

One  might  be  inclined  at  first  to  argue  that,  although  the  cost  of 
implementation  may  be  different  with  different  techniques,  the  nodal  performances 
for  Case  2  under  all  disciplining  regimes  can  be  made  virtually  the  same  provided 
all  other  factors  are  equal.  In  this  case  all  other  factors  are  basically 
concerned  with  local  clock  accuracy  at  the  point  in  time  that  the  last  remaining 
reference  was  temporarily  lost.  However,  further  considerations  indicate  that 
there  are  considerable  practical  differences  in  the  realization  of  equal 
performances  during  the  interval  of  Case  2  with  different  disciplining  techniques 
during  the  interval  of  Case  1.  For  example,  the  error  history  for  the  combination 
of  features  in  Run  16  would  be  a  much  simpler  function  than  that  for  the  features 
of  Run  3.  In  Run  16  this  history  has  been  reduced  to  essentially  the  performance 
of  the  local  clock  relative  to  the  network  master.  In  Run  3  the  network 
topology,  external  stress  events  and  individual  clock  performance  at  all  other 
nodes  of  the  network  affect  this  error  history.  Thus,  it  is  concluded  that  those 
features  which  reduce  the  complexity  of  this  error  history  during  the  interval  of 
Case  1  will  tend  to  increase  the  potential  for  survivability  during  the  interval 
of  Case  2. 
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3. 3. 6. 2  Case  1  -  References  Are  Available 

3. 3. 6. 2.1  Data  for  Evaluation 

The  method  for  evaluation  of  this  desirable  characteristic  during  the 
interval  of  Case  1  is  to  measure  frequency  errors  immediately  before  and  after 
dropouts  of  the  general  stress  scenario  runs  and  to  measure  the  ensemble  average 
of  nodal  frequency  error  variances  for  the  general  and  high  level  stress 
scenarios.  Tables  3. 3. 6. 2-1  through  3. 3. 6. 2-3  show  the  frequency  errors 
inmedi ately  before  and  after  the  dropouts.  Table  3. 3. 6. 2-4  shows  the  ensemble 
average  of  the  nodal  frequency  error  variance  during  the  duration  of  the  general 
and  high  level  stress  scenario  runs. 

3. 3. 6. 2. 2  Evaluation  of  Data 

•  Link  6-5  Failure 

Directed  single  ended  Runs  2G  and  13G  had  considerable  frequency 
error  due  to  link  variation  but  their  loop  integrators  remembered 
enough  to  go  considerably  closer  to  average  value. 

Directed  double  ended  Runs  8G,  11G,  and  15G  had  only  small  errors 
before  failure  and  all  experienced  increased  errors  upon  loss  of 
link . 

Directed,  double-ended,  ICEM&C  and  phase  reference  combining  Runs 
12G*  and  16G*  had  no  error  before  or  after  failure  because  they 
never  completely  lost  reference. 

None  of  the  mutual  control  runs  completely  lost  reference.  The 
double  ended  Runs  9G  and  10G  had  much  smaller  errors  immediately 
before  failure  of  the  link  than  any  of  the  single  ended  runs, 
roughly  1/20  that  of  Runs  3G,  4G,  and  7G  and  1/10  that  of  5G  and  6G. 
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Table  3. 3. 6. 2-1.  Frequency  Errors  Before  and  After  Link  6-5  Fails  at  150,000  Seconds 


trrors  on  Run  ?S  uere  stamped  by  link  variations  Runs  126*  and  166*  had  no  errors  before  or  after  event.  -  Too  small  to 
measure  In  relation  to  other  disturbances. 


Table  3. 3. 6. 2-3.  Frequency  Errors  Before  and  After  Node  13  Falls  at  250,000  Seconds 
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Table 

3. 3. 6. 2-4.  Frequency  Variances 

Run  Number 

General  Stress 
Scenario 

High  Level 
Stress  Scenario 

2  OC-2 

1.822x10”*° 

4.827x10”** 

3  MOEW 

3.034xl0”10 

8.878x10”** 

4  MC+UEW 

3.891xl0-10 

1.48x10'*° 

5  MC+M+EW 

1.594xl0'10 

5.067x10”** 

6  MC+M+UEW 

1.625xl0'10 

5.917x10-** 

7  MC+EW+OOS 

3.09xl0”10 

9.207x10”** 

8  DC+DE 

1.481x10"** 

1.652x10”** 

9  MC+DE+EW 

1.68x10”** 

1.893x10”** 

10  MC+M+OE+UEW+DOS 

1.31x10'** 

1.722x10”** 

11  DC+OE+ICEM&C 

1.433x10"** 

1.575x10”** 

11*  DC+DE+ICEM4C 

1.509x10’** 

1.467x10’** 

12*  DC+DE+ICEM&C+PRC 

1.464x10"** 

1.479x10"** 

13  DC+SO 

1.842x10”*° 

5.231x10”** 

14  OC+OE+SO 

1.39x10”** 

1.629.10"** 

15  DODE+ICEM&C+SO 

1.195x10'** 

1.376x10”** 

15*  X+DE+ICEM4C+S0 

1.32x10”** 

1.446x10"** 

16*  X+0E+ICEM4C+PRC+S0 

1.279x10”** 

1.478x10”** 

*  With  Jitter 


Link  1-2  Failure 
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The  errors  at  most  nodes  of  the  single  ended  runs  were  masked  by 
link  variations  on  Links  1-6  and  1-13. 

Errors  could  be  measured  for  most  runs  at  Nodes  2  and  3. 

Runs  12G  and  16G  had  no  measurable  errors  before  or  after  the 
failure. 

Mutual  runs  except  Run  10G  had  considerably  larger  errors  than 

directed  runs  irrmediately  before  failure  (5  to  10  times)  and  also 

had  larger  errors  iimiediately  after  the  failure.  Run  10G  had  small 

•  1  ? 

errors  before  and  after  failure  (Averages  of  1.3x10  before  and 
4.1xl0'12  after). 

Node  13  Fails  at  250,000  Seconds 

The  errors  of  Run  7G  were  masked  by  link  variations. 

Errors  for  the  directed  control  runs  were  confined  to  Nodes  16  and 
17  and  to  Node  16  for  Phase  Reference  Combining  runs. 

Errors  for  the  mutual  single  ended  runs  were  mostly  masked  due  to 
normal  link  variations  except  at  Nodes  15,  16,  and  17  and  at  node  2 
of  Runs  3G  and  4G. 

Average  errors  before  and  after  failure  were  as  follows: 


Before 

After 

Directed,  All 

3.06xl0-11 

4.38xlO'U 

Directed,  Double-Ended 

2.51xlO"U 

6.18xlO‘U 

Mutual,  All 

1.36xl0'10 

1 .41xl0"10 

Mutual,  Double-Ended 

2.51xlO'U 

5.41xlO’U 

•  Frequency  Variances 
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The  double-ended  feature  reduced  the  ensemble  average  of  frequency 
variances  more  than  any  other  feature. 

Addition  of  other  features  with  the  exceptions  of  unequal  reference 
weighting  and  dropout  smoothing  tended  to  reduce  the  variances  by 
small  amounts. 

3. 3. 6. 2. 3  Conclusions  About  Survivability  for  Case  1 

•  The  double-ended  feature  under  either  mutual  or  directed  control 
was  the  most  effective  feature  in  reducing  instantaneous  errors. 

•  The  directed  control  Run  16*  with  all  features  reduced 
instantaneous  errors  due  to  disturbances  at  all  nodes  more  than  any 
other  combination  of  features.  Thus,  this  combination  appears  to 
provide  a  more  survlvable  network  than  any  other. 

•  The  directed  control  Run  15  with  all  features  except  phase 
reference  combining  appeared  to  provide  the  second  most  stability 
and  accuracy.  This  combination  of  features  is  judged  to  provide 
the  second  most  survivable  network. 

•  Runs  10  and  14  were  both  judged  to  provide  the  third  most 
survivable  network. 

•  The  above  rankings  hold  when  combined  with  conclusions  for  Case  2. 
Further  rankings  are  given  in  the  overall  sumnary. 


2.  Independence  of  Clock  Error  Measurement  and  Correction 

3.  Phase  Reference  Combining 

4.  Self  Organization 

These  overhead  data  requirements  are  in  addition  to  the  requirements 
for  framing  synchronization  patterns  which  are  used  in  all  digital  data  multiplex 
systems  to  identify  the  beginning  of  major  groups  (frames)  of  data  bits  and  allow 
directing  the  bits  and  groups  of  bits  to  appropriate  demultiplexed  data  channels. 

In  a  system  solely  devoted  to  communications,  maintenance  of  bit  and 
framing  synchronization  is  sufficient  to  allow  uni  nterrupted  comuni  cat  ions  of  a 
channel  through  the  network.  Delays  through  the  network  will  not  cause  multiplexed 
channels  to  be  misrouted  as  long  as  both  bit  and  frame  sync  tracking  is 
maintained.  However,  if  delays  through  the  system  are  on  the  order  of  the 
duration  of  a  frame  or  larger,  and  it  is  desired  to  disseminate  precise  time, 
additional  less  frequent  unique  timing  markers  will  be  required  to  resolve 
ambiguities  in  frame  count.  Additional  requirements  on  overhead  data  needed  to 
disseminate  precise  time,  above  those  required  to  implement  bit  and  framing 
synchronization  are  discussed  in  Paragraph  3.3.8.  Paragraph  3. 3. 7. 6  discusses 
requirements  for  overhead  data  devoted  to  framing  synchronization. 
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is  shown  in  Figure  3. 3. 7. 1-1.  Here  Node  B  is  locking  to  a  reference  from  Node  A. 

The  phase  error,  measured  at  Node  A  Is  transmitted  to  Node  B  and 

utilized  to  correct  the  clock  at  Node  B.  This  phase  error  Information  is  shown  as 
being  filtered  by  F(S)  before  being  transmitted  to  reduce  overhead  requirements  as 
much  as  possible.  The  link  delay  is  denoted  by  r.  In  practice  this  would  be 
implemented  as  a  sampled-data  system,  but  for  the  purposes  of  this  discussion  it 
suffices  to  consider  the  analog  system. 

A  simplified  block  diagram  of  analog  form  is  derived  from  Figure 

3. 3. 7. 1- 1  by  means  of  block  diagram  transformations  and  is  shown  in  Figure 

3. 3. 7. 1- 2.  The  transfer  function  of  this  system  is 

<VS> 

Vs) 

For  loop  parameters  of  interest  in  DCS  synchronization,  the  link  delays 
are  very  short  relative  to  the  loop  time  constants  and  thus  may  be  neglected. 

Then,  assuming  r  =  0,  we  may  write 


The  overhead  data  transmission  is  the  sampled  output  of  the  filter  F(S).  In  order 
to  make  the  overhead  requirements  small,  one  must  reduce  the  bandwidth  of  F(S). 

The  approach  we  will  take  is  to  find  the  smallest  bandwidth  of  F(S)  that  can  be 
used  without  significantly  changing  the  transient  performance  of  the  loop  and  then 
estimating  the  overhead  requirements  for  this  filter.  It  turns  out  that  the 


e~s  (1  +  F( S) )  KH(S) 

2  $  +  KH( S)  (1  +  e'“T  F(S)) 


(3.3. 7-1) 
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requirements  are  so  small  that  the  choice  of  filter  bandwidth  is  not  critical. 

This  can  be  demonstrated  by  the  following  analysis  using  a  first  order  filter  F(S) 
*  b/(s  ♦  b) . 

For  a  Type  2  loop  (H(S)  =  (S  +  a)/S),  the  closed  loop  transfer  function 
(3. 3. 7-2)  becomes 

^  *  K(S  *a)  (S.Zb) - .  (3.3.7-3J 

2SJ  +  (K  +  2b)S^  +  K(a  +  2b)S  +  2Kba 


This  looks  quite  different  from  the  transfer  function  of  a  Type  2  loop, 
but  it  can  be  factored  to  give 

M, 


Vs) 


'1 


2S  +  (2b-k-2l<€)  +  K(1+*)S  +  Ka(l+«) 

For  F(S)  with  sufficiently  wide  bandwidth  e  and  6  both  approach  zero  and  the 
second  term  is  identical  to  the  transfer  function  of  the  Type  2  loop.  The  first 
term  has  a  time  constant  much  shorter  than  that  of  the  second  and  this  impacts 
transient  behavior  only  in  the  vicinity  of  t  =  0. 

The  quantities  €  and  6  are  given  by 
a  -  1  ♦  2e 


M2s  +  M3 


(3.3. 7-4) 


and 


where 


T"-1T  +  20 


1  +  2«  ,21 

2a-  (1  +  if)  =  -  7 


a  =  b/K 


+  2^  €(3  +  2«  -  2 a) 


(3.3. 7-5) 


(3. 3. 7-6) 


(3.3. 7-7) 


is  roughly  the  ratio  of  the  bandwidth  of  the  filter  F(S)  to  that  of  the  loop  (for 
large  $)•  The  first  term  in  Equation(3.3. 7-4)  has  a  time  constant 


-1  _  2d 
T  K(1  +  2«7 


(3.3. 7-8) 
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compared  with  r  =  1/K  for  the  type  two  loop  with  large  £  .  The  ratio  of  these  two 
time  constants  as  well  as  and  <  are  shown  in  Figure  3. 3. 7. 1-3  as  a  function  of  cr. 

We  -also  define  a  new  £  '  and  o/ n  for  the  second  term  of  Equation  (3. 3. 7. 1-4) 
which  are  related  to  the  original  loop  parameters  by 


^~-=\/r+S  (3. 3. 7-9) 


and 


£  '  1  +  e 

— = 


(3.3.7-10) 


These  ratios  are  also  shown  in  this  figure.  Note  that  for  a-  >  10  the 
system  behavior  is  almost  identical  to  that  of  the  Type  2  loop. 

To  get  the  required  overhead,  one  must  compute  the  required  sampling 
rate  for  the  output  of  this  filter.  It  is 


Ns  =  N  cr  K/2n- samples/second,  (3.3.7-11) 

where  N  is  the  number  of  samples  per  Hz.  In  practice  one  would  use  a  sharper 
rolloff  filter  to  reduce  N.  For  example,  a  fourth  or  fifth  order  Butterworth 
would  allow  one  to  use  an  N  in  the  range  of  10  to  20  and  obtain  less  than  1 
percent  interpolation  error.  (See  Davis  and  McRae  65  ).  This  is  compared  to  N 
=  10,000  for  the  first  order  filter.  Using  the  sharper  rolloff  increases  the 
settling  time  of  F(s)  by  about  a  factor  of  3  so  one  chooses  cr  >  30  for  this 
filter.  Conservatively  selecting  o-  =  50  and  N  =  20,  we  have  N$  = 
samples/second.  We  also  need  to  assure  that  sampling  rate  is  not  so  slow  that 
link  delays  are  significant  relative  to  the  loop  time  constant.  This  is  ensured 
if  Ns  »  K  which  is  satisfied. 

The  overhead  data  transmission  requirements  is  then 

Rd  =  Ns  Bs  (3.3.7-12) 
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where  B$  is  the  required  number  of  bits  per  sample  of  the  phase  error  waveform. 

The  required  B$  can  be  estimated  in  the  following  way.  The  worst-case  is  for 
the  quartz  clock  acquisition  mode.  Then  the  frequency  change  caused  by  a  phase 
step  is  given  by 

d  =  K4T  (3.3.7-13) 

with  6  measured  as  a  fractional  frequency  change  and  ^T  measured  in  seconds.  By 
letting  4T  represent  a  phase  quantization  interval  then  6  is  the  frequency  change 
observed  due  to  a  one  quantization  interval  change  in  phase.  If  we  select  6  = 
10-12,  then  that  frequency  change  will  be  negligible.  This  requires  that  4T  = 

1.0  ns.  Since  a  total  required  range  of  phase  error  is  less  than  10  ns  then  B$  = 
14  bits/sample  is  adequate.  Using  this  number  with  the  K  for  the  acquisition  mode 

gives  Rd  _  22  bits/second  which  is  slightly  more  than  1.5  samples/second.  To 
make  further  discussions  simpler  we  shall  assume  that  1.5  samples/second  are 

transmitted  giving  Rd  _  21  bits/second.  A  similar  number  could  be  obtained  for 
Type  1  loop  used  in  the  mutual  sync  approach.  If  precise  time  were  also  being 
distributed,  an  occasional  epoch  10  would  also  be  distributed  but  this  data 

requirement  would  be  much  smaller  than  Rd>  This  overhead  requirement  is  the 
same  as  that  of  distributing  INFO  3  in  the  improved  TRD  technique  as  discussed  in 
Paragraph  2.5. 

3. 3. 7.2  Independence  of  Clock  Error  Measurement  and  Correction 

This  feature  is  implemented  by  utilizing  an  overhead  channel  to 

distribute  the  estimate  of  each  node's  phase  error  relative  to  the  network  master 

to  all  neighboring  nodes.  This  allows  a  neighboring  node  which  is  deriving  its 

clock  from  that  node  to  obtain  an  estimate  of  the  error  of  its  clock  relative  to 
the  master  and  perform  the  correction  accordingly. 
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The  overhead  requirement  is  that  of  transmitting  a  phase  error  estimate 
to  each  neighboring  node  and  it  is  precisely  the  same  requirement  that  was 
discussed  in  the  previous  paragraph.  There  we  found  that  for  the  loop  parameters 
selected  that  only  21  bits/second  were  required.  Again,  this  data  is  identical  to 
that  called  INFO  4A  and  INFO  4B  in  Paragraph  2.2.5.  One,  but  not  both,  of  these 
would  be  transmitted  by  a  node  to  every  neighboring  node. 

3 . 3 . 7 . 3  Phase  Reference  Combining 

This  feature  described  in  Paragraph  2.2.4  reduces  the  contribution  of 
measurement  error  to  the  measured  timing  errors.  This  is  done  by  computing  a 
clock  error  with  respect  to  the  master  as  an  average  of  error  computed  over 
several  paths  to  the  master.  The  paths  are  carefully  chosen  so  that  there  are  no 
closed  loops  in  the  timing  hierarchy. 

The  references  available  via  the  multiple  paths  are  weighed  in  inverse 
proportion  to  the  estimated  variances  of  the  errors  over  each  of  these  paths. 

Thus,  in  order  to  implement  this  approach  each  node  must  pass  on  to  neighboring 
nodes  not  higher  in  the  timing  hierarchy  its  estimate  of  the  variance  of  the 
measured  but  uncorrected  error  at  that  node.  There  are  two  types  of  variances 
transmitted.  Using  Stover's  notation,  each  node  transmits  INFO  5A  to  all  nodes  at 
the  same  level  in  the  hierarchy  or  above.  Similarly,  It  transmits  INFO  5B  to  all 
neighboring  nodes  lower  in  the  hierarchy.  Thus,  there  is  a  requirement  to 
transmit  one  variance  estimate  on  each  overhead  link.  One  would  not  need  very 
many  quantization  levels  of  the  variance  in  order  to  achieve  near  optimum 
results.  Ten  bits  should  be  more  than  adequate  for  this  purpose.  This  data  would 
only  need  to  be  sent  very  infrequently  so  the  overall  data  rate  would  be  quite  low. 
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3.3. 7.4 
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Self-Organizing 

The  self -organ izing  feature  was  originally  proposed  by  Darwin  and 
Prim  ’  and  has  actually  been  implemented  in  the  Canadian  Data  Routed® 

In  Stover's  approach  two  types  of  data  must  be  transmitted 

between  neighboring  nodes  via  the  overhead  channel.  They  are  INFO  1  which  is  the 
rank  of  the  master  reference  for  the  local  clock.  Here  one  needs  only  enough  bits 
to  allow  a  unique  ID  for  all  nodes  in  the  network.  Ten  bits  would  allow  a  1024 
node  network.  Nine  bits  were  used  for  this  purpose  in  the  Canadian  Data  Route. 

The  second  type  of  data,  INFO  2,  is  the  total  number  of  links  between  the  local 
node  and  its  master.  Seven  bits  were  used  for  this  in  the  Data  Route,  and  this 
would  be  more  than  adequate. 

Thus,  we  see  that  a  total  of  17  bits  would  be  required  for 
se If -or gan iz ati on .  The  actual  data  rate  would  depend  on  how  frequently  this  data 
is  transmitted  which  in  turn  is  dictated  by  the  desired  speed  of  reorganization. 
Certainly  one  would  not  need  to  transfer  this  data  more  than  once  per  second. 

3. 3. 7. 5  Total  Requirements 

A  rough  estimate  of  the  total  overhead  requirements  for  the  features 
considered  can  now  be  made.  To  be  conservative,  we  will  assume  that  a  sample  of 
each  type  of  data  is  transmitted  in  each  overhead  word.  Because  of  the  data 
transmission  requirements  of  the  phase  error  estimates,  this  will  require 
transmission  of  1.5  overhead  words  per  second.  All  the  other  types  of  data  could 
be  transmitted  at  a  somewhat  slower  rate  than  this  while  maintaining  acceptable 
performance.  Also,  we  will  assume  that  each  message  will  be  three  times  the 
length  of  the  total  amount  of  data  to  allow  for  error  control  coding  and  a  framing 
word.  The  resulting  required  data  rates  are  shown  in  Table  3.3. 7.5. 
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Table  3. 3. 7. 5.  Overhead  Requirements  for  Synchronization  Features 


INFO  Number 

Feature 

Number  Bits 

Rate  (Bits/Second) 

1 

Self-organizing 

10 

45 

2 

Self-organizing 

7 

31.5 

3 

Double-ended 

14 

63 

4A  and  B 

Independence  of 
measurement  and 

error 

cor  recti  on 

14 

63 

5A  and  B 

Phase  reference 

combining 

10 

45 

Total 

55 

247.5 

Note  that  the  overhead  requirements  are  quite  modest.  By  comparison 
the  overhead  synchronization  channel  in  the  Canadian  Data  Route  requires  400 
bits/second. 

3.3. 7.6  Framing  Synchronization 

The  concept  of  framing  synchronization  is  very  simple.  A  fixed 
synchronization  pattern  of  N  bits  is  repeated  once  per  frame  to  denote  the 
beginning  of  a  new  frame.  The  pattern  may  be  N  consecutive  bits  or  may  be 
ditributed  among  the  data  bits.  The  length  of  the  pattern,  N,  may  be  1  or  more 
bits.  The  most  comnon  approach  to  frame  sync  acquisition  and  maintenance  in  the 
presence  of  noise  is  to  search  the  serial  data  with  a  pattern  recognizer.  Once  an 
acceptable  pattern  is  found  the  search  is  inhibited  and  the  periodic  reoccurrence 
of  the  pattern  is  tested  at  every  subsequent  expected  (frame)  interval. 

Willi ard  has  shown  that  for  a  given  synchronization  instrumentation  approach 


and  a  given  percentage  data  devoted  to  frame  sync  there  is  an  optimum  size  pattern 
(N)  and  spacing  between  patterns.  But,  spacing  between  patterns  must  logically  be 
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associated  with  frame  length.  Optimum  here  is  relative  to  minimum  time  to  acquire 
frame  synchronization.  Also  the  time  to  acquire  sync  and  the  time  to  detect  loss 
of  frame  sync  is  heavily  dependent  on  the  percentage  of  total  transmitted  bits 
which  are  devoted  to  a  frame  sync  pattern.  Many  modifications  and  refinements  to 
the  acquisition  and  maintenance  process  have  been  instrumental  or  suggested  which 
improve  performance  without  increasing  the  percent  data  devoted  to  sync.  However, 
the  complexity  of  hardware  is  usually  significant. 

The  T1  carrier  system  used  by  the  telephone  companies  transmits  at  1544 
kb/s  of  which  8  kb/s  or  0.52  percent  of  the  transmitted  bits  are  devoted  to 
framing  synchronization.  Many  other  multiplexing  equipments,  particularly  in 
military  applications  have  used  2  to  4  percent  of  the  data  space  to  transmit 
framing  sync  patterns.  This  amount  of  sync  information  is  necessary  when  sync 
acquisition  times  are  specified  to  be  under  100  milliseconds.  (50  milliseconds  is 
a  very  conrnon  specification.)  It  has  been  common  practice  to  specify  fast 
acquisition  times  and  this  is  almost  certainly  because  frequency  signal  fading  is 
expected  often  and  in  the  absence  of  very  accurate  nodal  clocks  it  is  not  expected 
that  bit  synchronization  will  be  maintained  during  fading  requiring  the  total 
reacquisition  of  bit  and  framing  synchronisation  following  each  signal  fading. 

The  first  significant  point  of  this  discussion  is  that  the  250  b/s 
required  to  implement  all  the  Time  Reference  Distribution  features  combined  as 
shOMi  in  Paragraph  3.3. 7. 5  is  only  1/32  of  the  amount  of  overhead  used  to  provide 
framing  synchronization  in  a  T1  carrier  system;  and  the  T1  carrier  rate  is  less 
than  1/20  of  the  expected  backbone  rate  to  be  used  in  the  DCS  network.  Thus  the 
amount  of  overhead  data  required  to  implement  all  the  sync  features  is  neglegible 
compared  to  the  overhead  which  will  typically  be  devoted  to  Frame  Synchronization. 
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3. 3. 7. 6.1  Reduced  Frame  Sync  Overhead  Requirements  when  Precise  Time  is 

Available  at  Nodes 

The  requirement  for  very  fast  sync  acquisition  time  cannot  be  logically 
defended  when  a  communications  system  is  first  turned  on  (startup).  Whether  a 
system  takes  50  mi  11  seconds  or  5  seconds  to  frame  synchronize  will  mean  nothing 
under  these  conditions  since  conriunicati ons  through  this  newly  turned  on  equipment 
cannot  begin  until  after  the  node  is  operational  and  the  initiation  of  traffic 
will  be  determined  by  human  response  time.  Therefore,  time  to  detect  loss  and 
reacquire  total  synchronization  must  be  dictated  by  dynamic  changing  conditions 
within  an  operating  network.  If  a  network  encounters  frequent  deep  fading  and 
nodal  clock  differnces  in  phase  are  continuously  shifting  then  most  fades  will  be 
accompanied  by  loss  of  both  bit  and  framing  synchronization  and  long  times  to 
acquire  sync  will  contribute  to  high  percentages  of  time  the  system  will  be  out  of 
synchronizati on. 

Now  consider  a  system  which,  once  bit  and  framing  synchronization  are 
initially  established,  will  never  lose  frame  synchronization,  even  in  the  absence 
of  signal  input,  provided  the  local  nodal  clock  does  not  slip  relative  to  the 
source  clock  during  the  fade  or  signal  outage.  A  system  of  this  type  was 
designed,  built  and  tested  by  Milliard^®.  This  system  contains  a  conventional 
search  and  verify  synchronizer  for  frame  synchronization.  However,  in  addition  a 
separate  frame  sync  mantainenance  circuit  controls  the  multiplexing  function. 

When  both  bit  and  frame  sync  is  lost  the  search  and  verify  circuit  finds  sync  and 

transfers  identification  of  the  true  sync  location  to  the  separate  frame  sync 

maintenance  circuit.  But  the  search  and  verify  circuit  is  locked  out  such  that  it 
can  never  look  for  frame  sync  in  the  same  location  being  monitored  by  the  frame 

sync  maintenance  circuit.  Thus  when  a  signal  loss  or  fade  occurs,  the  maintenance 

circuit  continues  to  monitor  the  last  known  sync  location  (if  the  bit  rate  clock 
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does  not  drift  more  than  a  bit  interval)  based  on  clock  pulses  advancing  the  DEMUX 
counters.  During  the  fade  or  signal  loss  the  search  and  verify  circuit 
continuously  looks  for  better  framing  sync.  When  signal  returns  the  maintenance 
circuit  is  still  in  sync  (if  no  clock  drift  slip  occurs).  If  a  clock  drift  slip 
does  occur  then  the  search  and  verify  circuit  finds  the  new  location  and  transfers 
it  to  the  maintenance  circuit. 

Use  of  such  a  dual  sync  system  in  conjunction  with  a  system  in  which 
the  nodal  clocks  did  not  drift  more  than  a  bit  period  during  most  signal  fades  or 
dropouts  would  virtually  eliminate  the  time  a  communication  link  is  out  of  sync 
due  to  fades  and  short  term  outages. 

This  approach  would  significantly  reduce  the  requirements  for  fast 
acquisition  time  and  thus  a  system  with  the  greatest  phase  stability  at  nodes 
would  need  much  less  overhead  data  devoted  to  framing  synchronization  patterns. 
3.3.8  Precise  Time  Availability 

If  certain  features  are  included  in  a  network  synchronization  approach, 
the  approach  may  very  easily  be  adapted  for  dissemination  of  precise  time.  In 
this  paragraph  we  will  discuss  those  combinations  of  features  which  are  required 
for  providing  precise  time.  Additionally,  the  disadvantages  of  implementing  this 
are  discussed. 

3. 3. 8.1  Dissemination  of  Precise  Time 

In  this  paragraph  we  will  discuss  the  features  which  facilitate  the 
dissemination  of  precise  time.  Precise  time  can  be  distributed  by  any  system  that 
is  double-ended  and  has  directed  control  (this  includes  mutual  sync  if  the  network 
is  referenced  to  a  master).  These  are  the  only  features  actually  necessary  for 
distribution  of  precise  time.  Other  features  such  as  "independence  of  error 
measurement  and  correction"  and  "phase  reference  combining"  may  provide  improved 


accuracy  in  the  distribution  of  precise  time,  but  they  are  not  necessary  to 
accomplish  it.  We  will  show  that  one  can  take  a  network  synchronization  approach 
using  these  features  that  does  not  provide  precise  time  and  with  a  slight 
modification  make  provision  for  disseminating  precise  time.  Furthermore,  this  can 
be  done  without  altering  in  any  way  the  manner  in  which  the  nodal  frequency  is 
produced  by  the  nodal  synchronizer.  Thus,  there  is  absolutely  no  penalty  in  terms 
of  network  synchronizer  performance  that  is  incurred  as  a  result  of  requiring  the 
system  to  provide  precise  time. 

This  result  can  be  illustrated  by  considering  the  system  shown  in 
Figure  3. 3. 8. 1-1.  We  assume  a  double-ended  system  with  Node  B  slaving  to  Node  A. 
The  times  of  the  clocks  are  denoted  by  Ta  an(j  ancj  the  link  delays  are  Dab 
and  Dba.  In  addition,  the  phase  errors  measured  at  Nodes  A  and  B  are  denoted 
by  ^ab  and  0ba,  respectively.  In  a  double-ended  system  without  precise  time 
the  phase  locked  loop  (PLL)  deriving  the  nodal  frequency  at  Node  B  would  be  driven 
by  the  estimated  phase  error. 

fl  ba  »  Aba  ~  ^ab  (3. 3. 8-1) 


This  process  is  very  similar  to  that  used  in  a  precise  time  system  such  as 
TRD.170  One  starts  by  computing  apparent  timing  errors  at  each  node  (through 
transmission  of  time-of-day  beacons). 
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The  form  of  these  is  closely  related  to 

and  ^ba  except  for  the  inclusion  of  the 
also  be  written  as 


(3. 3. 8-2) 
(3. 3.8-3) 

the  computation  of  the  phase  errors  0ab 
path  delay  term.  This  timing  error  can 
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ATa  =  cab  -  Dba  (3. 3. 8-4) 

where  «ab  is  the  actual  timing  error  between  Clocks  A  and  B.  One  could  also 
write  ATa  in  terms  of  the  output  of  the  phase  detector  for  the  double-ended  phase 

detector  system  0ab>  1<e.f 

ATg  =  0ab  -  aa.  (3. 3. 8-5) 

where  aa  chosen  to  compensate  for  the  absolute  difference  in  the  phase  error, 
#ab,  measured  at  Node  A  and  the  apparent  timing  error  at  Node  A,  Thus, 

"a  =  0ab  "  ATa  (3. 3. 8-6) 

The  measured  phase  error  has  an  ambiguity  of  some  arbitrary  number  of  cycles. 
However,  once  0ab  ancj  relationship  toATa  are  defined  then«a  is  a 
constant  and  any  changes  inATg  cause  corresponding  changes  in  o&b- 

The  typical  precise  time  system  would  cause  the  clock  at  Node  B  to  be 
corrected  based  on  the  calculated  timing  error. 

«: .  *ATb-ATa  (3. 3. 8-7) 

Dd  - 2 - 

which  by  using  Equation  (3.3. 8-4)  and  the  fact  that  Dba  =  _Dab  gives 

f  ba  =  eba  +  Pba  '  Dab  (3. 3. 8-8) 

Thus,  the  correction  term  in  Equation  (3. 3. 8-8)  is  equal  to  the  actual  timing 
error  plus  a  link  delay  term  which  vanishes  if  Dba  equais  gab . 

The  relationship  between  TRD  and  a  double-ended  system  can  be 
demonstrated  by  using  Equation  (3. 3. 8-7)  and  Equation  (3. 3. 8-5)  to  write 

<  l.  =  ^ba  -  flab  +  “a  -  «  b  (3. 3. 8-9) 

ba  - 2 -  — J - 

Thus,  we  see  that  the  additional  correction  term  (aa  .  ab)/2  is 

required  to  convert  a  normal  double-ended  system  into  a  precise  time  system.  This 

correction  term  is  a  constant  and  can  be  estimated  through  very  infrequent 
transmissions  of  a  time-of-day  mark.  This  would  allow  continual  refinement  of  the 
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estimate  of  «a  and«b.  During  the  interim  period  continual  changes  in  the 
estimate  «ba  WOuld  be  made  through  changes  in  0ba  and  0ab- 

However,  one  does  not  have  to  drive  the  nodal  synchronizer  PLL  with  the 

estimated  phase  error  €ba  as  determined  from  Equation  (3. 3. 8-9).  An 
alternative  and  potentially  better  implementation  would  allow  separation  of  the 
nodal  synchronizer  function  from  the  precise  time  function.  One  could  implement 
the  phase  locked  loop  and  thus  the  clock  correction  just  as  it  would  be 
implemented  in  a  normal  double-ended  system. 

This  implementation  is  shown  in  Figure  3. 3. 8-2.  The  loop  is  driven  by 
the  same  phase  detector  estimates  0ab  an(j  #ba  that  one  woui d  have  used  without 
precise  time.  Thus,  the  nodal  frequency  f„  behaves  just  as  in  a  normal 
double-ended  system.  Time  of  day  is  given  by  counting  cycles  of  this  clock  after 
suitable  initialization.  If  the  precise  time  references  used  in  all  phase 
comparisons  were  the  uncorrected  output  of  the  counter,  then  the  clock  correction 
term  would  be  the  static  phase  offset  of  the  local  precise  time  counter  which  is 

Cu*=«a'“b  (3.3.8-10) 

d  2 

This  is  the  second  term  in  «ba  as  given  in  Equation  (3. 3. 8-8).  The  first 
term  which  drives  the  PLL  is 


0 , 


'  Jba  '  *ab 


(3.3.8-11) 


e  ba  =*ba  +  C*b 


7ba  "  T 
Thus»  'ba  can  be  written  as 

(3.3.8-12) 

Note,  however,  as  we  have  shown  in  Figure  3. 3. 8-2,  that  the  precise 
time  comparisons  can  be  made  on  the  corrected  time  indication.  In  this  case  each 
new  clock  correction  term  must  be  added  to  the  previous  one  and  the  additional 
terms  will  approach  zero  as  t — «-«.  The  correction  term  at  time  i  then  becomes 
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Figure  3. 3. 8-1.  Double-Ended  System  for  Disseminating  Precise  Time 
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Figure  3. 3. 8-2.  Implementation  of  a  Nodal  Synchronizer  Providing  Precise  Time 
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Cb  (i)  =  Cb  (1_1)  +  C*b  (3.3.8-13) 

where  C*b^j  g^ven  by  Equation  (3.3.8-10). 

When  the  features  of  "Independence  of  phase  error  measurement  and 
correction"  and  "phase  reference  combining"  are  added,  the  clock  correction  will 
be  done  in  a  similar  fashion  but  the  term  is  slightly  more  difficult  to  compute. 

To  simplify  the  following  discussion,  let  us  denote  the  correction  term  at  Node  B 

as  given  in  Equation  (3.3.8-10)  by  Cba  to  indicate  that  it  is  the  time  offset 
relative  to  the  clock  at  Node  A.  Then  in  implementing  the  feature  "independence 
of  phase  error  measurement  and  correction"  with  precise  time,  the  clock  correction 
term  at  Node  B  would  be 


€  .  +  € 
ba  am 


(3.3.8-14) 


In  this  way  the  clock  at  Node  B  would  be  corrected  based  on  an  error  measurement 
relative  to  the  master  rather  than  relative  to  Node  A.  Then  using  Equations 
(3.3.8-11)  -  (3.3.8-13),  Equation  (3.3.8-14)  can  be  written  as 


f  bm  ^  ba  +  ^  am  +  ^bm* 


(3.3.8-15) 


The  clock  correction  term  in  this  case  is 

^bm  =  Cba  +  Cam  •  (3.3.8-16) 

In  a  similar  fashion,  the  feature  "phase  reference  combining"  can  also 

be  added.  This  results  in  forming  a  clock  correction  term  based  on  the  weighed 
sum  of  several  available  references.  If  Node  B  has  available  N  references  with 
paths  back  to  the  master,  it  will  form  the  clock  error  estimate 


€  bm~  i=i  Wi  ^  e  bi+  €  im^ 


(3.3.8-17) 
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(3.3.8-18) 
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The  clock  correction  term  is 


Wi 


(C 


bi 


+■  C.  ) 
im' 


(3.3.8-19) 


Thus,  Equation  (3.3.8-18)  shows  that  once  again  the  process  can  be  separated  into 
a  form  whereby  a  term 


N 

&  "i 

which  would  drive  a  PLL  and  make  nodal  frequencies  track  properly  and  a  term 
which  is  just  a  clock  correction  term  needed  to  provide  precise  time.  This 
demonstrates  that  there  can  be  no  performance  penalty  associated  with  the 
inclusion  of  precise  time  in  a  system  since  the  nodal  frequencies  are  determined 
precisely  as  they  would  be  in  a  system  without  precise  time. 

3. 3. 8. 2  Penalties  for  Inclusion  of  Precise  Time 

There  are  no  significant  penalties  associated  with  including  the 
precise  time  capability  in  the  network  synchronization  subsystem  and  there  are 
significant  benefits.  As  we  pointed  out  in  the  previous  paragraph,  there  is  no 
performance  penalty  associated  with  providing  precise  time  because  the  phases  and 
frequencies  to  the  nodal  clocks  are  controlled  in  exactly  the  same  manner  as  they 
would  be  in  a  system  without  precise  time.  Thus,  phase  and  frequency  accuracy  are 
the  same.  There  are  several  other  potential  benefits.  The  dissemination  of 
precise  time  will  result  in  improved  system  monitor abi 1 ity.  When 
self-reorganization  is  used,  the  improved  monitorabi 1 ity  will  result  in  faster 
response  time  for  reorganization  which  in  turn  improves  survivability.  The 
provision  of  precise  time  may  also  improve  flexibility  to  the  extent  that  future 
DCS  users  that  can  make  use  of  precise  time  will  have  it  readily  available. 
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The  only  two  penalties  associated  with  precise  time  are  increased  cost 
and  overhead  requirement,  but  in  both  cases  the  impact  is  insignificant.  As  shown 
in  the  block  diagram  of  Figure  3. 3. 8-2,  additional  costs  will  be  incurred  by 
adding  a  counter,  a  suimer,  and  some  additional  processing  to  the  system. 

However,  the  processing  requirement  is  extremely  small  compared  to  the  other 
processing  requirements  in  the  nodal  synchronizer  because  it  need  be  done  rather 
infrequently  relative  to  the  other  processing.  Thus,  the  processing  could  be  done 
through  time-sharing  the  nodal  synchronizer  processor.  The  counter  and  sinner 
represent  additional  hardware  but  it  represents  an  insignificant  increment  to  the 
total  hardware  in  the  nodal  synchronizer.  Additional  overhead  is  required,  but  as 
we  pointed  out  in  the  preceding  paragraph,  the  sampling  range  for  the  precise  time 
overhead  could  be  made  a  small  fraction  of  total  overhead  depending  upon  the 
desired  accuracy.  Thus,  the  impact  on  overhead  transmission  requirements  could  be 
made  negligible. 

3.3.9  Synchronization  Subsystem  is  Monitorable 

The  characteristic  of  monitorability  is  the  level  to  which  the  timing 
subsystem  can  provide  data  used  in  assessing  its  own  performance.  It  is  apparent 
that  monitorability  is  valuable  at  least  to  the  extent  that  an  option  such  as  a 
redundant  piece  of  equipment,  an  alternate  link  or  another  timing  source  is 
available  and  can  be  utilized  as  the  result  of  decisions  made  on  the  basis  of 
monitored  information. 

Monitorability  is  a  significant  characteristic  because  it  can  lead  to 
greater  availability  through  providing  tolerance  to  failure.  If  potential 
failures  can  be  identified  prior  to  failures  actually  occurring,  these  situations 
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can  be  remedied  In  an  off-line  basis  or  more  rapidly  by  knowing  the  nature  of  the 
failure  before  it  occurs.  The  result  is,  therefore,  both  an  increase  in  MTBF  and 
a  reduction  in  MTTR  and  therefore  a  greater  availability. 

It  would  be  desirable  to  actually  calculate  the  increase  in  timing 
subsystem  availability  due  to  the  increased  monitorabi 1 ity  afforded  by  each 
feature.  However,  this  is  impractical  to  do  in  this  study  since  it  requires 
exhaustive  analysis  of  a  detailed  system  design.  What  we  can  do,  though,  is  to 
identify  the  unique  information  provided  for  the  monitoring  function  by  each 
feature  and  to  assess  the  utility  of  this  information. 

The  nodal  synchronizer  monitoring  function  should  collect  data  to  allow 
detection  of  two  types  of  failures: 

•  Local  nodal  synchronizer  hardware  failures  or  impending  failures. 

•  Failures  or  impending  failures  at  other  nodes  in  the  network. 

This  data  is  used  to  make  decisions  on  appropriate  actions  to  be  taken  by  the 
synchronizer  such  as  switching  to  redundant  hardware  elements  for  local  failures 
and  switchover  to  alternate  references  for  failures  at  other  nodes.  This  , 

switching  could  be  done  either  manually  or  automatically  depending  upon  the 
desired  speed  with  which  this  action  should  be  accomplished.  In  addition, 
significant  events  detected  by  the  monitoring  function  should  be  reported  to  the 
Tech  Control  Element  and  may  eventually  be  reported  to  other  nodes. 

Any  timing  approach  will  utilize  a  collection  of  data  for  monitoring 
purposes  which  is  independent  of  the  specific  features  used  in  that  approach. 

This  data  includes,  but  is  not  limited  to,  various  hardware  status  monitoring 
signals  from  the  modems  and  the  nodal  synchronizer  processor,  phase-locked  loops. 
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and  clocks.  These  signals  typically  give  an  indication  of  a  failure  after  it 
occurs  and  is,  of  course,  very  useful.  In  fact,  the  bulk  of  the  monitoring 
function  will  depend  on  this  data.  However,  certain  synchronization  features 
provide  additional  data  which  is  useful  not  only  in  detecting  failures,  but  also 
in  locating  synchronization  subsystem  functional  elements  that  have  degraded  or 
are  in  the  process  of  failing.  Thus,  monitorabi lity,  and  hence  availability,  can 
be  enhanced  by  using  the  unique  aspect  of  several  of  the  features  to  determine 
out-of-specification  trends,  predict  failures,  and  to  isolate  these  faults.  In 
the  following  paragraphs  we  will  identify  the  unique  information  provided  for  the 
monitoring  function  by  each  feature  and  to  assess  the  utility  of  this  information. 
3.3. 9.1  Directed  Control 

The  information  that  is  available  for  monitoring  in  a  directed  control 
system  is  the  reference  phase  error.  This  is  the  phase  error  between  the  local 
clock  and  the  incoming  reference.  It  will  be  available  for  the  link  currently 
being  referenced  and  also  possibly  for  all  other  potential  references.  Since  a 
Type  2  loop  can  be  used  in  a  directed  control  system,  this  phase  error  should  be 
zero  in  steady  state.  Thus,  large  excursions  of  phase  error  indicate  potential 
problems  either  with  the  local  clock  or  with  one  of  the  clocks  in  the  timing 
distribution  chain  between  the  local  node  and  the  master.  Knowledge  of  this  data 
from  several  different  references  would  allow  the  monitoring  function  to  decide 
whether  the  problem  is  with  the  local  clock  or  an  external  clock  and  to  take 
appropriate  action.  This  data  from  each  node  could  also  be  reported  to  an  overall 
System  Control  for  possible  localization  of  external  faults. 
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There  is  a  problem,  though,  in  utilizing  this  information.  On  many 
links,  and  particularly  on  satellite  links,  there  are  large  path  delay  variations 
which  are  reflected  in  the  measured  phase  error  for  a  single-ended  system.  Thus, 
the  actual  contribution  of  degraded  elements  such  as  clocks  to  the  measured  phase 
error  may  be  undetectable  until  they  get  very  large.  Thus,  there  is  much  less 
utility  in  anticipating  degraded  conditions  with  this  data  than  if  the  path  delay 
variation  effects  were  removed.  For  this  reason  we  rate  the  utility  of  this  data 
only  as  fair.  This  approach  for  using  this  data  is  consistent  with  the  way  in 
which  it  is  used  for  status  monitoring  on  the  AT&T  Digital  Data  System. 

3. 3. 9. 2  Mutual  Control 

In  a  single-ended  system  employing  mutual  control  the  same  data  is 
available  as  for  directed  control.  That  is,  we  can  monitor  the  measured  phase 
errors  of  all  references  utilized.  One  can  draw  basically  the  same  conclusions  as 
in  the  previous  paragraph  as  to  how  the  data  would  be  utilized  with  one 
exception.  That  is  that  since  there  are  no  distinct  timing  distribution  paths  it 
would  be  more  difficult  for  an  overall  System  Control  to  use  this  data  to  localize 
an  impending  malfunction.  In  addition  to  this  problem,  the  task  of  determining  a 
degraded  condition  is  made  more  difficult  by  the  fact  that  the  mutual  control 
system  must  use  Type  1  loops  which  track  a  frequency  offset  with  a  nonzero 
steady-state  phase  error.  Thus,  typically  each  of  the  references  is  at  a  large 
steady-state  phase  error,  and  there  are  also  the  normal  phase  error  variations  due 
to  path  delay  variations.  Because  of  this  the  actual  contribution  to  measured 
phase  error  caused  by  degraded  elements  must  get  much  larger  to  be  detected  than 
it  must  be  in  a  directed  control  system.  Therefore  there  is  even  less  utility  in 
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using  this  data  to  anticipate  degraded  conditions.  For  this  reason  we  rate  the 
utility  of  this  data  as  poor. 

3. 3.9. 3  Double-Ended 

When  either  a  directed  control  or  a  mutual  control  system  is  made 
double-ended,  the  phase  error  of  the  local  clock  relative  to  its  reference  is 
computed  in  such  a  fashion  that  the  effects  of  path  delay  variations  are 
eliminated.  This  makes  this  data  very  useful  in  a  directed  control  system  since 
the  path  delay  variations  are  the  main  factor  degrading  the  utility  of  measured 
phase  errors.  Thus,  as  phase  errors  get  large,  they  very  rapidly  indicate  that 
either  the  local  clock  or  a  clock  higher  in  the  timing  distribution  hierarchy  is 
degrading.  Therefore  we  rate  the  utility  of  this  data  in  a  double-ended  directed 
control  system  as  good. 

In  a  mutual  control  system  the  double-ended  feature  removes  the  effect 
of  path  delay  variations  on  phase  error,  but  this  is  not  the  main  factor  degrading 
the  utility  of  the  measured  phase  errors.  The  principle  degrading  factor  is  the 
use  of  Type  1  loops  which  result  in  large  nonzero  steady-state  phase  errors. 
Because  of  this  the  actual  contribution  to  measured  phase  error  caused  by  degraded 
elements  still  must  get  very  large  in  order  to  be  detected.  For  this  reason  the 
double-ended  feature  adds  very  little  to  the  overall  utility,  and  we  rate  the 
utility  of  this  data  in  a  double-ended  mutual  control  system  as  poor. 

3. 3. 9. 4  Independence  of  Clock  Error  Measurement  and  Correction 

This  feature  is  considered  for  a  double-ended  directed  control  system. 
Though  one  does  not  necessarily  have  to  use  the  double-ended  feature  with  this 
feature,  it  does  not  make  much  sense  not  to  use  it.  The  same  phase  errors 
computed  for  a  double-ended  system  are  available.  In  addition,  the  phase  error  of 
all  adjacent  nodes  relative  to  the  master  are  known.  Thus,  the  phase  error  used 
to  correct  the  local  clock  is  the  phase  error  measured  relative  to  the  network 
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master.  Having  this  information  adds  a  great  deal  of  utility  in  isolating  a 
potential  problem.  Clock  perturbations  higher  in  the  timing  distribution 
hierarchy  do  not  affect  this  measurement  so  a  large  increase  in  phase  error 
relative  to  the  master  will  indicate  either  a  problem  with  the  local  clock  or  the 
network  master.  Appropriate  action  can  then  be  taken.  Reporting  this  data  to 
System  Control  will  be  helpful  in  anticipating  problems  with  the  network  master 
clock.  Thus,  we  rate  utility  of  this  data  as  very  good. 

3. 3. 9. 5.  Phase  Reference  Combining 

This  feature  is  considered  in  combination  with  the  features  discussed 
in  the  preceding  paragraph.  The  data  available  with  that  combination  of  features 
is  also  available  in  this  case  and  can  be  used  in  the  same  way.  The  additional 
data  provided  by  this  feature  is  a  set  of  estimated  inaccuracies  of  the  phase 
errors  measured  at  all  adjacent  nodes  relative  to  the  master.  Properly  combining 
these  phase  errors  will  allow  some  improvement  in  the  estimated  phase  error  of  the 
local  clock  relative  to  the  master.  This  is  accomplished  by  weighting  the 
contribution  of  each  available  reference  inversely  with  its  estimated  variance. 
While  this  will  provide  slightly  more  accuracy  than  the  combination  of  features 
discussed  in  the  preceding  paragraph,  it  would  not  result  in  a  significant 
increase  in  monitorabi lty.  Thus,  we  continue  to  rate  the  utility  of  this  data  as 
very  good. 

3. 3. 9. 6  Self-Organizing 

The  self-organizing  feature  in  conjunction  with  directed  control,  can 
be  combined  with  each  of  the  other  features  discussed  in  the  preceding 
paragraphs.  It  provides  varying  degrees  of  additional  utility  depending  upon  the 
combination  of  features.  The  self-organizing  feature  provides  data  about  the 
current  topology  of  the  timing  distribution  network,  and  it  allows  fast  switchover 
to  alternate  references  in  the  event  of  real  or  anticipated  malfunctions.  The 
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information  about  the  network  topology  is  very  helpful  since  any  reorganization 
activity  indicates  trouble  in  the  network.  In  addition,  the  ability  to  quickly 
switch  to  an  alternate  reference  is  provided  and  is  actually  essential  to  take 
advantage  of  the  data  provided  by  the  other  features  to  anticipate  degraded 
conditions.  For  this  reason  the  utility  of  the  data  provided  by  the  other 
features  when  combined  with  the  self-organizing  feature  will  be  upgraded.  As  a 
result  we  rate  the  utility  of  these  features  as:  directed  control  -  good, 
double-ended  -  very  good,  independence  of  phase  error  measurement  and  correction  - 
excel  lent,  and  phase- reference  combining  -  excellent. 

3. 3. 9. 7  Summary 

We  would  like  to  emphasize  that  the  overall  monitorability  of  any 
timing  technique  that  is  implemented  can  be  made  good.  However,  what  we  have  been 
discussing  is  the  additional  utility  provided  by  individual  features.  A  summary 
of  these  results  is  given  in  Table  3. 3. 9. 7.  Also,  we  show  our  rating  of  the 
overall  monitorability  of  a  system  employing  each  of  these  features.  In  most 
cases  the  additional  data  will  not  significantly  affect  the  overall 
monitorability.  However,  we  believe  that  the  monitorability  will  be  significantly 
improved  if  the  Self -Organizing  feature  is  used  in  combination  with  either 
Independence  of  Phase  Error  Measurement  and  Correction  or  Phase  Reference 
Combining.  This  is  indicated  in  the  table. 
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Table  3. 3. 9. 7.  Utility  of  Data  Provided  by  Features 
for  the  Monitorability  Function 


Utility  of  Overall 


Feature 

Additional  Data 

Monitorabi 1 ity 

Directed  Control 

Fair 

Good 

Mutual  Control 

Poor 

Good 

Double-Ended  Directed  Control 

Good 

Good 

Double-Ended  Mutual  Control 

Poor 

Good 

Independence  of  Phase  Error 
Measurement  and  Correction 

Very  Good 

Good 

Phase  Reference  Combining 

Very  Good 

Good 

Self  Organizing  with: 

Directed  Control 

Good 

Good 

Double-Ended 

Very  Good 

Good 

Independence  of  Phase 

Error  Measurement  and 
Correction 

Excellent 

Very  Good 

Phase  Reference  Combining 

Excellent 

Very  Good 
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3.3.10  Compliance  With  Federal  Standard  1002 

3.3.10.1  Requirements 

Federal  Standard  1002  states  that  "the  time  and  frequency  reference 
information  utilized  in  applicable  Federal  Government  teleconmunications 
facilities  and  systems  shall  be  referenced  to  (know  in  terms  of)  the  existing 
standards  of  time  and  frequency  maintained  by  the  U.S.  Naval  Observatory, 
UTC(USNO),  or  the  National  Bureau  of  Standards  UTC(NBS)."  It  is  herein 
interpreted  that  insofar  as  the  DCS  network  synchronization  subsystem  is 
concerned,  the  above  statement  implies  some  form  of  comparison  or  calibration  of 
the  frequency  and  time  generating  elements  of  the  DCS  network  in  order  that  time 
and  frequency  signals  of  the  DCS  network  may  be  referenced  to  (known  in  terms  of) 
UTC(L)SNO)  or  UTC(NBS).  Federal  Standard  1002  further  states  that  "the  accuracy  of 
this  time  and  frequency  reference  information  with  respect  to  UTC(USNO)  or 
UTC(NBS)  shall  be  comnensurate  with  the  individual  system  design  and  interface 
requirements."  It  is  herein  interpreted  that  this  latter  statement  means  that  how 
closely  the  DCS  network  synchronization  subsystem  time  and  frequency  signals  need 
be  maintained  to  those  of  UTC(USNO)  or  UTC(NBS)  depend  on  the  accuracy 
requirements  of  the  DCS  network,  as  well  as  any  interface  requirements  between  the 
DCS  network  and  externa  1  systems  including  the  interface  between  the  DCS 
synchronization  subsystem  and  the  UTC(USNO)  or  UTC(NBS)  reference  time  and 
frequency  signals.  Thus,  the  system  design  parameters  of  the  DCS  synchronization 
subsystem  dictate  how  often  the  DCS  synchronization  subsystem  time  and  frequency 
generating  sources  must  be  compared  and/or  calibrated  against  the  UTC(USNO)  or 
UTC(NBS)  reference  signals. 

3.3.10.2  Impact  of  the  Standard  on  the  Features 
3.3.10.2.1  Directed  Control 

If  the  DCS  synchronization  subsystem  utilizes  the  directed  control 


feature,  it  is  quite  easy  to  comply  with  Federal  Standard  1002  because  in  this 

case  only  the  DCS  network  master  clock  and  its  alternates  need  the  capability  to 
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be  calibrated  against  the  UTC(USNO)  or  UTC(NBS)  reference  signals.  If  the 
UTC(USNO)  or  UTC(NBS)  time  and  frequency  reference  signals  are  available 
continuously  at  the  DCS  network  master  clock  site,  then  the  simplest  thing  to  do 
may  be  to  slave  the  DCS  network  master  clock  to  these  reference  signals  by  means 
of  a  phase-lock  loop  or  other  equivalent  control  method.  In  case  the  UTC(USNO)  or 
UTC(NBS)  reference  signals  are  not  continuously  available  at  the  DCS  network 
master  clock  site,  then  some  periodic  calibration  may  be  accomplished.  One  such 
method  is  to  use  a  portable  clock.  In  either  case  the  impact  of  the  requirements 
of  Federal  Standard  1002  is  minimal  on  the  DCS  network  in  the  sense  that  only  the 
master  clock  and  its  alternates  in  the  DCS  network  need  be  calibrated  if  the 
directed  control  feature  is  utilized. 

3.3.10.2.2  Mutual  Control 

The  mutua.  control  feature  without  a  master  may  pose  some  problem  in 
compliance  with  Federal  Standard  1002.  The  difficulty  arises  in  that  this  feature 
does  not  provide  means  whereby  either  the  system  frequency,  individual  nodal 
frequencies  or  the  timekeeping  devices  derived  from  these  frequencies  may  be 
periodically  calibrated  against  the  UTC(USNO)  or  UTC(NBS)  signals.  Without  such 
periodic  calibration  no  method  could  be  devised  which  would  guarantee  to  a  user 
that  the  network  synchronization  signals  would  be  maintained  within  some  specified 
tolerance  of  the  UTC(USNO)  or  UTC(NBS)  time  and  frequency  reference  signals.  On 
the  other  hand  if  the  mutual  control  feature  incorporates  a  master  then  only  this 
master  clock  and  its  alternates  need  be  calibrated  against  the  UTC(USNO)  or 
UTC(NBS)  reference  signals.  The  same  comments  as  were  made  for  the  directed 
control  feature  would  then  be  applicable  provided  a  method  of  selecting  the 
alternate  were  provided. 

3.3.10.2.3  Double  Ended 

In  order  that  synchronization  subsystems  which  include  the  double  ended 


feature  be  in  compliance  with  Federal  Standard  1002  it  may  be  necessary  that  some 

subunits  of  the  network  synchronization  subsystem  be  located  at  the  physical  site 
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of  the  USNO  and/or  NBS  master  reference  clocks.  The  DCS  network  synchronization 
equipment  at  the  USNO  or  NBS  master  reference  site  would  in  some  fashion  be  slaved 
to  the  UTC(USNO)  or  UTC(NBS)  reference  signals  and  would  include  the  ability  to 
make  the  two-way  exchange  of  information  with  the  highest  level  nodes  of  the  DCS 
network  synchronization  subsystem.  Thus  making  synchronization  subsystems  which 
utilize  the  double  ended  feature  compliant  with  Federal  Standard  1002  is  dependent 
on  whether  or  not  the  required  equipment  can  be  collocated  at  the  USNO  and/or  NBS 
site. 

3.3.10.2.4  Self-Organizing 

If  the  self -organizing  feature  is  employed  and  a  new  master  is  selected 
then  this  new  master  may  need  to  be  calibrated  against  the  UTC(USNO)  or  UTC(NBS) 
reference  signals  if  it  remains  the  network  master  for  an  appreciable  period  of 
time.  Thus,  the  UTC(USNO)  or  UTC(NBS)  reference  signals  must  be  available  at  all 
nodes  which  may  be  automatically  selected  for  the  master,  if  compliance  with 
Federal  Standard  1002  is  to  continue  following  reorganization.  These  requirements 
also  hold  for  mutual  systems  utilizing  a  master. 

3.3.10.2.5  Overhead  Channel 

If  the  DCS  network  synchronization  subsystem  is  to  be  compliant  with 
Federal  Standard  1002  then  some  channel  is  required  from  the  UTC(USNO)  or  UTC(NBS) 
site  to  node(s)  of  the  DCS  network  whether  it  be  a  portable  link,  a  telephone 
channel,  an  HF  radio  link,  or  a  satellite  link.  In  this  sense  some  overhead 
channel  is  required  because  its  only  purpose  is  to  carry  the  UTC(USNO)  or  UTC(NBS) 
time  and  frequency  reference  signals  to  appropriate  nodes  of  the  DCS  network. 

3.3.10.2.6  Independence  of  Clock  Error  Measurement  and  Correction 

This  feature  is  not  directly  impacted  by  Federal  Standard  1002  but 
inclusion  of  this  feature  allows  for  smaller  tolerances  in  the  deviation  of  time 
and  frequency  signals  of  the  DCS  network  synchronization  subsystem  from  those  of 
UTC(USNO)  or  UTC(NBS). 
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3.3.10.2.7  Phase  Reference  Combining 
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This  feature  is  not  impacted  by  Federal  Standard  1002  except  that  the 
accuracy  of  the  DCS  network  synchronization  subsystem  signals  may  be  improved  by 
its  inclusion  and  hence  the  DCS  network  synchronization  subsystem  signals  may 
deviate  less  from  those  of  UTC(l)SNO)  or  UTC(NBS). 

3.3.11  Cost-Effectiveness  of  Features 

The  directed  control  and  mutual  control  features  do  not  require  an 
overhead  channel.  All  other  features  of  interest  require  an  overhead  channel. 

The  bit  rates  required  in  the  overhead  channel  to  implement  each  of  the  features 
are  quite  modest,  as  shown  in  Paragraph  3.3.7.  Since  a  digital  system  is  being 
implemented  it  may  safely  be  assumed  that  a  basic  set  of  hardware  at  each  node 
associated  with  the  synchronization  subsystem  will  include  at  least  a 
microcomputer.  Such  a  computer  is  sufficient  to  run  a  digital  phase  locked  loop 
which  may  be  used  in  implementing  one  of  the  disciplined  techniques  (directed 
control  or  mutual  control).  It  is  quite  probable  that  this  computer  may  also  be 
capable  of  implementing  one  or  more  of  the  additional  features.  If  this  is  true 
then  the  major  cost,  aside  from  the  overhead  channel,  of  implementing  the 
additional  features  will  be  nonrecurring  engineering  costs  for  programming  the 
microcomputer  to  perform  these  additional  features.  First,  let  us  see  how  much 
processing  time  and  memory  space  will  be  required  for  disciplining  the  local  clock. 

3.3.11.1  Running  the  Basic  Control  Loop 

3.3.11.1.1  Directed  Control 

Figure  3.3.11.1.1-1  shows  a  block  diagram  of  a  phase-iocked  loop 
employing  a  proportional  plus  integral  control  scheme  (Type  2  loop).  Figure 
3.3.11.1.1-2  shows  the  digital  apparatus  sufficient  to  implement  the  control 
scheme  of  Figure  3.3.11.1.1-1.  This  apparatus  consists  of  a  phase  comparator,  an 
arithmetic  processor  (microprocessor)  and  a  local  clock.  The  local  clock  is 
assumed  to  consist  of  a  d/a  converter  and  divider  chain.  If  the  local  oscillator 
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Figure  3.3.11.1.1-1.  Block  Diagram  of  Phase-Locked  Loop  Using  Type  2  Loop 
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Diagram  of  Local  Clock  Frequency  Digital  Controller 
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is  cesium  controlled  then  an  outboard  microphase  stepper  is  assumed.  If  the  local 
oscillator  is  quartz  controlled  then  it  is  assumed  to  be  directly  voltage 
controlled  from  the  output  of  the  d/a  converter.  The  phase  comparator  is  a 
start-stop  counter  which  compares  the  phase  epoch  of  the  reference  signal  with 
that  of  the  local  clock.  The  phase  epoch  of  the  reference  signal  may  be  obtained 
from  the  framing  pulses  of  the  incoming  communications  link.  For  purposes  of 
discussion  let  us  assume  that  these  pulses  occur  at  a  basic  4  kHz  rate.  Thus,  the 
phase  comparator  is  of  the  sawtooth  type  and  the  phase  interval  within  which  the 
local  clock  will  be  controlled  is  +125  pS.  We  will  further  assume  that  the  local 
clock  signal  occurs  at  a  2.56  MHz  rate.  Then  the  counter  of  the  phase  comparator 
must  be  able  to  hold  at  least  2.56  x  106  x  250  x  10-6  =  640  counts  of  the 
local  clock  signal.  The  interval  0-320  indicates  that  the  local  clock  is  retarded 
in  phase  from  the  reference  whereas  the  interval  from  321  through  640  indicates 
that  the  local  clock  is  advanced  in  relation  to  the  reference  signal.  A  10  bit 
binary  counter  is  sufficient  for  holding  the  maximum  count.  Sufficient  samples  of 
the  phase  difference  should  be  averaged  to  obtain  a  good  estimate  of  the  actual 
phase  difference,  i.e.,  equipment  jitter  such  as  the  +1  count  error  of  the 
start-stop  counter  should  be  smoothed  out,  but  the  averaging  period  should  be 
short  in  relation  to  other  time  constants  of  the  system  because  this  determines 
the  sampling  rate,  1/r  of  the  digital  control  loop.  Due  to  system  time  constants, 
e.g.,  the  interval  for  initially  coming  on  line  with  a  set  of  acquisition  mode 
loop  parameters,  a  sampling  rate  as  high  as  1.5  times  per  second  may  be  required 
(see  Paragraph  3.3.7).  With  requal  to  667  ms  approximately  2667  samples  of  the 
phase  error  can  be  obtained  for  averaging.  If  N  is  chosen  as  a  power  of  2  the 
counter  and  averaging  device  can  be  combined  into  a  single  longer  counter.  The 
more  significant  digits  can  then  be  taken  as  the  averaged  sample.  The  arithmetic 
processor  forms  a  proportional  plus  integral  correction  signal  to  be  applied  to 
the  local  clock.  It  performs  a  multiplication  by  a  fixed  constant  (1/M)  and  two 
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addition  operations  once  each  sampling  period.  It  also  inputs  the  averaged  phase 
error  estimate  and  outputs  the  P  bit  control  signal  to  the  local  clock.  The  fixed 
constant  1/M  of  Figure  3.3.11.1.1-2  times  the  update  frequency  1/r  corresponds  to 
the  integrator  constant  a  of  Figure  3.3.11.1.1-1.  This  constant  determines  how 
fast  the  integrator  signal  will  build  or  decay.  The  multiplying  constant  K  of 
Figure  3.3.11.1.1-1  is  a  gain  factor  which  determines  how  fast  the  local  clock 
phase  will  respond  to  the  composite  proportional  plus  integi  al  control  signal.  In 
the  diagram  of  Figure  3.3.11.1.1-2  this  term  is  embodied  in  the  amount  that  the 
control  signal  is  allowed  to  swing  the  local  clock  frequency,  e.g.,  the  composite 
control  signal  would  have  to  reach  a  level  that  is  inversely  proportional  to  the 
size  of  this  term  before  the  local  clock  frequency  would  change  some  specified 
amount  from  its  natural  value.  Since  the  phase  of  the  clock  is  equal  to  the 
integral  of  the  clock  frequency,  a  time  constant  is  involved  with  the  gain 
factor  K. 

If  the  local  clock  is  a  hign  quality  quartz  clock  it  may  have  a 
fractional  natural  frequency  drift  of  1  x  10-10  per  day.  Thus  it  may  be  desired 
to  allow  the  maximum  value  of  the  control  signal  to  change  the  clock  frequency  by  a 
maximum  fractional  amount  of  1  x  10-6  in  order  to  operate  without  mechanical 
readjust  for  the  life  of  the  equipment.  If  the  P  bit  control  signal  is  chosen  such 
that  the  least  significant  bit  will  change  the  local  clock  frequency  by  a  fractional 
amount  of  3.1  x  10-H  then  P  should  be  approximately  Log2  (io-6/3.1  x  10-H)  = 

Lo92  32000  a  15  bits.  Since  the  125  ps  phase  comparator  control  interval  is 
broken  up  into  320  counts  of  the  start-stop  counter  and  each  count  corresponds  to  a 
change  of  the  least  significant  bit  of  the  P  bit  control  signal  or  a  fractional 
frequency  change  of  3.1  x  10-H  the  value  of  K  will  then  be  (3.1  x  10-H) 
x  (1/125  x  10-6)  x  (320)  =  7.94  x  10-5  sec-1. 

If  the  local  clock  is  a  cesium  clock  then  the  values  of  a  and  K  would  be 
chosen  different  from  that  of  the  quartz  clock  in  order  to  take  advantage  of  its 
inherently  higher  accuracy. 
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If  the  value  of  N  of  Figure  3.3.11.1.1-2  is  chosen  to  be  2ll  then  21 
bits  will  hold  all  the  significance  of  the  averaged  error  sample.  The  micro¬ 
computer  then  needs  only  to  input  three  8-bit  bytes  each  667  ms  and  store  this 
information  in  the  temporary  storage  area.  If  the  time  constant  (1/a)  of  the 
integrator  is  made  equal  to  3  days  then  the  multiplying  constant  (1/M)  of  Figure 
3.3.11.1.1-2  is  1/(1. 5  x  3  x  86400)  =  1/(388800)  «  2-18.6.  Thus  it  appears  that 
it  is  sufficient  to  perform  all  the  arithmetic  operations  using  binary  lengths  of 
32  bits  (four  8-bit  bytes). 

In  order  to  estimate  the  time  and  storage  space  required  for  an  Intel 
8080A  microprocessor  to  execute  the  directed  control  loop,  four  byte  arithmetic 
subroutines  were  written  and  a  machine  language  program  was  coded  to  perform  the 
necessary  operations  using  these  subroutines.  No  claim  is  made  that  the  program 
or  subroutines  are  optimal  nor  that  they  would  actually  perform  correctly  but  such 
an  exercise  provides  sufficient  tools  for  obtaining  fairly  accurate  estimates. 
Table  3.3.11.1.1-1  shows  processor  time  and  memory  space  required  for  the 
subroutines.  Applicable  portions  of  this  information  will  be  used  in  the 
remainder  of  this  section  as  required.  The  directed  control  loop  main  program 
required  460  ps  per  iteration  and  105  bytes  of  storage  space.  Added  to  this  is 
3800  +  2  x  350  +  2  x  80  =  4660  ps  for  one  multiply,  two  adds,  one  input  and  one 

output.  Thus  the  total  processor  time  is  5120  ps/iteration.  This  represents 

5.12  x  10-3  x  1.5  x  100%  =  0.768%  of  the  microprocessor's  time  at  the  1.5 
iterations  per  second.  The  total  memory  space  is  395  bytes. 

The  multiply  time  used  in  the  above  analysis  could  be  reduced  by  at 

least  a  factor  of  two  simply  by  choosing  a  later  generation  microprocessor  such  as 
the  Zilog  Z-80  which  is  capable  of  being  clocked  at  a  4  MHz  rate.  The  multiply 
time  could  also  be  reduced  dramatically  by  choosing  the  multiplying  constant  1/M 
to  be  a  power  of  2.  In  this  case  the  multiply  operation  may  be  accomplished  by 
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Table  3.3.11.1.1-1.  Microprocessor  Multibyte  Subroutine  Time  and 

Space  Requirements 


Subroutine 

Processor  T ime 

i<s 

Space 

(Bytes) 

4  byte  by  4  byte 
multiply 

3800 

170 

4  byte  by  4  byte 
add 

350 

60 

4  byte  input 

80 

30 

4  byte  output 

80 

30 

4  byte  divide  by  2 

95 

60 

4  byte  integer  divide 

15203 

100 

4  byte  by  4  byte 

350 

60 

subtract 


shifting.  A  third  method  for  reducing  the  multiply  time  to  essentially  the  time 
necessary  to  output  the ‘quantities  to  be  multiplied  together  and  input  the  result 
back  into  the  microprocessor  is  to  provide  a  hardware  multiplier.  Nevertheless, 
even  with  the  software  multiply  routine  and  arbitrary  value  of  1/M,  very  little  of 
the  microprocessor's  time  is  required  to  run  the  directed  control  loop.  Thus 
approximately  99  percent  of  the  microprocessor's  time  will  be  available  for 
performing  duties  involved  with  other  features. 

3.3.11.1.2  Mutual  Control 

Figure  3.3.11.1.2-1  shows  a  block  diagram  of  a  Type  1  phase-locked 
loop.  Figure  3.3.11.1.2-2  shows  a  block  diagram  of  a  digital  implementation  of 
the  loop  filter  of  Figure  3.3.11.1.2-1.  The  multiplying  term  a  can  be  lumped  with 
a  scale  factor  K.  In  order  to  Incorporate  such  a  loop  into  a  mutual  control 
scheme  n  reference  signals  must  be  combined  and  then  applied  to  this  loop.  These 
operations  are  shown  in  Figure  3.3.11.1.2-3.  An  8080A  machine  language  program 
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INPUT  WEIGHT  COMBINE 


FILTER 


<t>9y  «<T) 


4#  {kT) 


W. 


0^,  (kT| 


OUTPUT 

OUTPUT 
CONTROL 
SEQUENCE 
TO  LOCAL 
CLOCK 


10066-11 


Figure  3.3.11.1.2-3.  Digital  Implementation  of  Mutual  Control  Weighting, 

Combining,  and  Loop  Filter 
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has  been  coded  to  perform  the  operations  of  this  figure.  The  main  program 
requires  (248  ♦  160  n)  /jS  per  iteration  and  (124  +  8  n)  bytes  for  data  and 
program.  Added  to  this  is  3800  (n  +  1)  ♦  350  n  ♦  80  (n  ♦  1)  /is  for  running  the 
multibyte  multiply,  add  and  I/O  subroutines.  Since  this  is  a  basic  disciplining 
technique  we  must  also  add  390  bytes  for  holding  these  subroutines.  Thus  the 
total  time  and  space  required  is  as  follows: 

Time  =  4128  +  4390  n  j/s/iteration 

Space  =  514  +  8  n  bytes, 
where  n  is  the  number  of  neighbors. 

For  n  ■  10  and  a  loop  iteration  rate  of  1.5  times  per  second  this  amounts  to 
approximately  7.2  percent  of  the  microprocessor ' s  time  and  requires  594  bytes  of 
storage. 

3.3.11.2  Double  Ended 

The  double  ended  feature  removes  clock  phase  offset  that  is  due  to  link 
delay.  Thus,  instead  of  deriving  the  reference  signal's  phase  epoch  at  the 
receiving  end  of  the  communications  link  it  is  effectively  derived  from  the 
sending  end  of  the  link  when  the  double  ended  feature  is  employed.  The  following 
discussion  will  explain  in  a  simple  manner  how  this  may  be  accomplished.  A  real 
system  probably  would  not  be  implemented  exactly  in  this  manner  because  a  real 
system  would  probably  be  constrained  by  having  a  dedicated  low  rate  overhead 
channel  for  transferring  the  necessary  information.  Nevertheless,  the  basic  idea 
can  be  obtained  from  the  following  discussion: 

Suppose  node  8  obtains  its  reference  from  node  A. 

1.  Node  B  transmits  his  time  Tfal  t0  node 

2.  After  transmission  delay  Dba  nocje  A  receives  B's  transmission, 
notes  his  own  clock  Tal  and  calculates  Kfl  =  Tal  +  Dba  -  Tbl. 


2838A 


3-148 


3.  At  a  short  time  later,  node  ^  transmits  the  quantities  Ta2 
*n<1  Ka  to  node  8. 

4.  After  transmission  delay  Dab  no(je  g  receives  A's  transmission, 
notes  his  own  clock  Tb2,  and  calculates  Kb  =  Tb2  +  Dab  -  Ta2. 

5.  Node  B  then  calculates  the  quantity 
‘Ka-Kb>'2*  |Tal  *  °ba  -  Tbl  -  <Tb2  *  °ab  *  Ta2>] « 

*  lTal  -  Tbl>  *  <Ta2  -  Tb2>  *  <°ba  '  °ab>|/2 
Now  since  the  clocks  are  assumed  to  drift  very  little  relative  to  each  other 
during  the  exchange  interval  (Tb^  to  Tb2),  the  quantity  j ( -  Tbj)  +  (Ta2  -  Tb2)|  /2 

should  be  a  good  estimate  of  Ta  -  Tb.  The  quantity  (Dba  -  Dab)/2  is  one-half  the 
asynmetry  in  the  two-way  transmission  delay.  Again,  if  the  exchange  interval 
(Tbl  to  Tb2)  is  small  then  this  term  is  expected  to  be  quite  small.  Thus 

Ta  -  Tb  S  (Ka  -  Kb)/2  .  (3.3.11.2-1) 

3.3.11.2.1  Calculations  for  Directed  Control 

The  phase  error  term  of  Equation  (3.3.11.2-1)  can  be  used  directly  in 
place  of  the  phase  difference  signal  of  the  single  ended  scheme  to  drive  the 
digital  phase-locked  loop.  Figure  3.3.11.2.1-1  shows  a  flow  chart  of  the 
processing  operations  sufficient  to  implement  the  double  ended  feature  that  are 
above  and  beyond  those  for  single  ended  directed  control.  An  8080A  machine 
language  program  has  been  coded  that  is  sufficient  for  implementing  this  flow 
chart.  The  main  program  requires  407+337n  ns  per  iteration  and  156+12n  bytes  of 
storage  space.  The  main  program  calls  the  four  byte  subtracter  n+1  times,  the 
four  byte  divide  by  2  once,  and  the  input  and  output  subroutines  n  times  each. 

This  results  in  an  additional  time  and  space  requirement  of  455+510n  ^s  per 
iteration  and  60  bytes  of  storage  space  for  a  total  of  952+887n  ns  per  iteration 
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Figure  3.3.11.2.1-1.  Flow  Chart  of  Processing  Operation  for 
Double  Ended  Feature  Under  Directed  Control 
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and  216+12n  bytes  of  storage.  With  10  neighbor  nodes  and  an  iteration  rate  of  1.5 
times  per  second,  this  results  in  the  following  percentage  of  processor  time  and 
bytes  of  space: 


••  ■ 


%  Time  1.47* 

Space  336  bytes 

3.3.11.2.2  Calculations  for  Mutual  Control 

An  8080A  microprocessor  program  has  been  coded  to  perform  the 
double  ended  feature  under  mutual  control.  The  main  program  requires  61+845n  ^s 
per  loop  iteration  and  24n+151  bytes  of  storage  space.  Calls  to  the  four  byte 
subroutines  consumes  1385n  jis  per  iteration.  Sixty  bytes  of  storage  space  is  also 
required  to  hold  the  four  byte  divide  by  2  subroutine.  Thus  a  total  of  61+2330n  /xs 
per  iteration  and  211+24n  bytes  of  storage  is  required  over  and  above  that  for 
mutual  control  with  single  ended.  For  n=10  neighbors  and  an  iteration  rate  of  1.5 
times  per  second  this  amounts  to  the  following  space  and  percent  processor  time: 

Processor  Time  3.345% 

Space  451  bytes 

3.3.11.3  Independence  of  Clock  Error  Measurement  and  Correction 

This  feature  removes  the  effects  of  clock  phase  error  occuring  at 
nodes  between  the  ultimate  master  and  the  node  in  question.  Thus,  each  node  in 
each  chain  keeps  up  with  its  own  measured  but  uncorrected  error  relative  to  that 
of  the  network  master  and  passes  this  error  term  on  to  its  neighbors. 

3.3.11.3.1  Processor  Operations  Under  Directed  Control 

Under  the  directed  control  regime  the  local  node  must  input  the 
error  of  its  immediate  reference  neighbor  with  respect  to  the  network  master 
node.  The  local  node  must  then  add  this  value  to  the  error  of  the  local  clock 
phase  with  respect  to  its  immediate  reference  node  to  obtain  its  own  error  with 
respect  to  the  network  master  node.  The  local  node  must  then  output  this  error 
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term  to  any  neighbor  nodes  lower  in  the  hierarchy.  An  8080A  machine  language 
program  has  been  written  to  perform  these  operations.  The  main  program  requires 
669  ns,  per  iteration  and  71  bytes  of  storage  space.  It  calls  the  four  byte  add 
subroutine  and  the  input  and  output  subroutines  once  each  iteration.  Thus  a  total 
of  1179  /is  per  iteration  and  71  bytes  of  storage  is  required.  This  represents 
0.1768  percent  of  the  processor  time  at  1.5  iterations  per  second. 

3.3.11.3.2  Processor  Operations  Under  Mutual  Control 

The  independence  of  clock  error  measurement  and  correction  feature 
cannot  be  applied  to  a  mutual  control  system  because  of  the  feedback  loops  which 
are  inherent  to  mutual  control. 

3.3.11.4  Phase  Reference  Combining 

This  feature  attempts  to  reduce  measurement  error  by  taking  advantage 
of  statistical  averages  over  a  number  of  paths  from  the  ultimate  reference  to  each 
node.  This  is  somewhat  similar  to  a  theorem  in  statistics  which  states  that  if  X 
is  a  normally  distributed  variate  with  mean  /j and  variance  o ^  and  a  random  sample 
of  size  n  is  drawn,  then  the  sample  mean  f  will  be  normally  distributed  with  mean  n 
and  variance  a  / n.  The  problem  at  hand  does  not  exactly  fit  this  theorem 
because  the  sample  of  size  n  from  the  n  paths  is  not  necessarily  random  (the  paths 
may  not  be  completely  independent  of  each  other)  and  each  member  of  the  sample 
does  not  necessarily  come  from  the  same  random  variate  nor  is  each  random  variate 
necessarily  normally  distributed  with  the  same  mean  and  variance.  Nevertheless, 
Stover  concludes  that,  by  choosing  suitable  weighting  factors  for  the 
signals  received  over  the  various  parallel  paths,  a  combined  signal  can  be  formed 
whose  variance  is  equal  to  the  parallel  sun  of  error  variances  of  the  n  parallel 
paths,  i.e.,  the  variances  add  like  resistors  in  parallel.  Thus,  implementation 
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Figure  3.3.11.3.2-1.  Functional  Diagram  of  Nodal  Processing  for 
Independence  of  Clock  Error  Measurement  and 
Correction  Using  Mutual  Control 
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of  this  feature  requires  knowing,  at  each  node,  how  to  weight  the  signals  received 


over  the  various  paths  in  order  to  reduce  the  variance  of  the  combined  error 
signal  to  the  minimum.  Stover's  weighting  strategy  involves  weighting  each  signal 
inversely  proportional  to  the  variance  in  measurement  error  of  each  parallel 
path.  The  weighting  factors  are  also  normalized  so  that  their  sum  is  equal  to  1 
at  each  node.  Two  sets  of  error  measurement  and  inaccuracy  information  are  formed 
at  each  node  based  on  input  information,  information  stored  at  the  node  concerning 
link  parameters,  and  self-information  of  the  node.  The  two  sets  of  information 
differ  depending  on  whether  the  input  information  was  obtained  from  neighbor  nodes 
that  are  higher  in  the  hierarchy  than  the  local  node  (Class  1  output  information) 
or  whether  the  input  information  was  obtained  from  neighbor  nodes  that  are  not 
lower  in  the  hierarchy  than  the  local  node  (Class  2  output  information).  The 
Class  1  information  is  used  to  correct  the  local  clock  and  is  passed  to  neighbor 
nodes  not  lower  in  the  hierarchy  than  the  local  node.  The  Class  2  information  is 
passed  to  all  neighbor  nodes  that  are  lower  in  the  hierarchy  than  the  local  node. 
These  rules  for  combining  the  information,  using  it  at  the  local  node,  and  passing 
it  to  neighbor  nodes  provide  for  maximum  utilization  of  information  while  at  the 
same  time  avoids  closed  loops. 
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In  addition  to  making  the  two-way  exchange  of  information  described  in 
Paragraph  3.3.11.2  to  implement  the  double-ended  feature  and  thereby  obtain  the 
difference  in  the  local  clock's  reading  with  respect  to  each  of  its  neighbors  with 
transient  effects  removed,  the  basic  processing  concerned  with  the  phase  reference 


combining  feature  involves  the  following: 

Input: 

1.  Class  1  error  measurements 

2.  Class  1  inaccuracy  (estimated  variance  of  1.) 


3.  Class  2  error  measurements 

4.  Class  2  inaccuracy  (estimated  variance  of  3.) 


Compute: 


1.  Class  2  output 

2.  Class  1  output 


3.  Class  2  output 

4.  Class  2  output 


error  measurements 
inaccuracy 


error  measurement 
inaccuracy 


Output: 

1.  Class  1  error  measurements 

2.  Class  1  inaccuracy 


3.  Class  2  error  measurements 

4.  Class  2  inaccuracy 


from  neighbor 
nodes  higher 
in  hierarchy 

from  neighbor 
nodes  not 
lower  in 
hierarchy 


computed 
using  input 
Class  1 
information 

computed 
using  input 
Class  2 
information 


to  neighbor 
nodes  not 
1 ower  i n 
hierarchy 

to  neighbor 
nodes  lower 
in  hierarchy 
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It  is  anticipated  that  the  error  measurement  information  (both  Class  1 
and  Class  2)  will  be  passed  through  the  network  at  the  basic  loop  iteration  rate 
but  it  is  not  believed  that  the  inaccuracy  information  (estimated  variances  of  the 
Class  1  and  Class  2  error  measurement  information)  need  be  updated  at  such  a  high 
rate  because  this  type  of  information  will  remain  fairly  constant  over  much  longer 
periods  of  time  than  the  loop  iteration  interval.  Factors  which  affect  the 
inaccuracy  information  are  basic  design  parameters  associated  with  the  links, 
nodal  measuring  equipment,  thermal  noises  in  the  control  loops  and  short  term 
jitter  and  drift  rates  of  the  nodal  frequency  source.  An  example  of  variance  in 
nodal  measuring  equipment  is  a  time  delay  measurement  device  which  uses  a 
start-stop  counter  in  conjunction  with  a  high  frequency  oscillator.  Such  a  device 
has  a  built-in  jitter  of  +1  period  of  the  high  frequency  oscillator.  It  also  has 
a  variance  term  associated  with  short  term  and  long  term  inaccuracy  of  the 
oscillator.  Another  source  of  the  inaccuracy  term  is  the  variance  in  delay 
through  up/down  converters  and  IF  strips  associated  with  the  transmitting  and 
receiving  equipment  at  each  node.  These  variance  terms  are  functions  of 
temperature,  power  supply  voltages,  component  aging,  and  other  factors.  The 
inaccuracy  terms  must  also  be  updated  each  time  there  is  a  failure  of  a  node  or 
link  of  the  network  in  order  to  reflect  any  resulting  change  in  the  number  and 
quality  of  reference  sources  at  each  node.  Although  it  is  not  presently  known 
precisely  how  often  the  inaccuracy  information  should  actually  be  updated,  it  will 
be  assumed  in  the  remainder  of  this  section  that  it  will  be  updated  once  per  hour. 

Table  3.3.11.4-1  shows  permanent  storage  tables  associated  with  the 
phase  reference  combining  feature.  The  microprocessor  inputs  the  Class  1  and 
Class  2  error  measurement  information  once  each  loop  iteration  period.  It  also 
performs  the  two-way  exchange  of  information  with  its  neighbors  to  obtain  the  time 


Table  3.3.11.4-1 


MEASURED  BUT  UNCORRECTED 
ERROR  OF  nl  NEIGHBOR  NODES 

CLOCKS  WITH  RESPECT  TO  / 
NETWORK  MASTER  WHO  ARE  X 
HIGHER  IN  HIERARCHY  THAN 
LOCAL  NODE 


MEASURED  BUT  UNCORRECTED 
ERROR  OF  n2  NEIGHBOR 
NODES  CLOCKS  WITH  RESPECT  / 
TO  NETWORK  MASTER  WHO  ARE  N 
NOT  LOWER  OR  HIGHER  IN 
HIERARCHY  THAN  LOCAL  NODE 


TIME  DIFFERENCES 
BETWEEN  LOCAL  CLOCK 
AND  NEIGHBOR  NODES 
WITH  TRANSIENT 
TIMES  REMOVED 


MEASURED  BUT 
UNCORRECTED 
ERROR  OF  LOCAL  / 
CLOCK  WITH  S 
RESPECT  TO 
NETWORK  MASTER 


Oata  Involved  With  Phase  Reference  Combining 


COMPUTED  AND  OUTPUT  DATA 
FOR  ERROR  AND  VARIANCE  OF 
LOCAL  CLOCK  WITH  RESPECT 
TO  NETWORK  MASTER 


difference  between  the  local  clock  and  its  neighbors  with  transient  effects 
removed  as  explained  in  Paragraph  3.3.11.2.  It  then  calculates  the  Class  1  and 
Class  2  error  measurements  for  output  and  correction  of  the  local  clock  each  loop 
iteration  period  using  the  Class  1  and  Class  2  weighting  factors.  The  input 
variances  may  be  updated  at  any  time  during  each  1  hour  period  since  this 
information  is  only  used  to  calculate  the  weighting  factors  once  each  hour  or  each 
time  a  network  configuration  change  takes  place.  The  output  Class  1  and  Class  2 
variances  need  be  calculated  only  once  per  hour  or  following  a  network 
reorganization. 

3.3.11.4.1  Calculating  the  Class  1  and  Class  2  Error  Measurements 

Figure  3.3.11.4.1  shows  a  flow  chart  of  the  processing  sufficient  for 
calculating  the  Class  1  and  Class  2  output  error  measurements  using  the  data  shown 
in  Table  3.3.11.4-1.  These  calculations  are  performed  once  each  loop  iteration 
interval . 

3.3.11.4.2  Calculating  the  Output  Variances 

The  Class  1  and  Class  2  output  variances  may  be  obtained  from  the 
following  formula: 


a 


2 


(3.3.11.4.2-1) 


where 


is  the  input  variance  from  reference  i 


aLi2  is  a  variance  term  associated  with  the  link 

between  the  local  node  and  reference  node  i. 


is  a  variance  associated  with  equipment  at  the 
local  mode.  It  may  be  obtained  by  periodically 
applying  special  test  sequences  at  the  input 
to  the  local  node  and  then  measuring  the  results. 


2838A 


3-158 


Figure  3.3.11.4.1.  Flow  Chart  For  Calulating  Class  1  and  Class  2  Output 

Measurement  Error 


3.3.11.4.3  Calculating  the  Weighting  Factors 

The  weighting  factors  may  be  obtained  form  the  following  formula: 


i  "i*  +  °L1Z 

1  .  'n  f 

k-1  +  o’LK2 


+  trLiZ 

I - 

-n - Y 

-  o-N 


(3.3.11.4.3.3-1) 


2  2 

Where  rr  and  <rN  are  defined  in  Equation  (3.3.11.4.2-1). 

3.3.11.4.4  Microcomputer  Time  and  Space 

A  machine  language  program  has  been  coded  for  performing  the  phase 
reference  combining  feature  as  described  in  the  previous  subparagraphs.  The 
microprocessor  time  and  storage  space  f<>r  performing  the  operations  concerned  with 
this  feature  are  as  follows: 

Once  per  loop  interation 

949  +  5692 (NL)  +  4511(H)  »  882  (NH)  /is  per  loop 
iteration 
Once  per  hour 

17570  +  35754(NL)  +  20655(H)  ps  per  hour 
Storage  Space: 

Data  -  46  +  32(NL)  +  4(H)  +  12((NH)  bytes 
Program  -  491  bytes 
Subroutines  -  220  bytes 

where  NL  is  the  number  of  neighbor  nodes  not  lower  in  the  hierarchy  than  the  local 
node,  H  is  the  number  of. neighbor  nodes  higher  in  the  hierarchy  than  the  local 
node  and  NH  is  the  number  of  neighbor  nodes  not  higher  in  the  hierarchy  than  the 
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local  node.  With  the  number  of  neighbors  equal  to  ten  and  H=3,  NL=6,  NH=7  and  a 
loop  iteration  rate  of  1.5  times  per  second  this  results  in  the  following  percent 
of  microprocessor  time  and  storage  space: 

X  Time  -  8.22  +  0.0082  3  8.23* 

Space  -  334  +  491  +  220  3  1045  bytes 
3.3.11.5  Self-Organizing 

The  self-organizing  scheme  described  in  this  section  is  similar  to  that 

ppo 

proposed  in  c  .  It  is  different  from  that  proposed  by  Darwin  and 

pic 

Priur1  in  that  after  the  selection  of  the  master,  the  rest  of  the  network  is 
arranged  on  the  basis  of  connectivity  rather  than  on  information  that  is  exchanged 
between  nodes. 

The  self-organizing  scheme  described  here  requires  that  each  node  of 
the  network  be  given  a  rank  that  is  in  accordance  with  its  order  of  succession  to 
the  master.  Further,  each  node  must  transmit  to  its  neighbor  nodes  the  rank  of 
its  ultimate  master  reference  node  and  the  number  of  nodes  between  itself  and  its 
master  reference.  Upon  entering  the  network,  a  node  must  reference  itself  until 
it  obtains  enough  samples  of  the  above  information  to  be  sure  that  there  are  no 
transients  in  the  information  which  would  cause  the  local  node  to  choose  a  bad 
reference.  This  self-organizing  scheme  assumes  that  the  desired  network  hierarchy 
is  relatively  static  and  that  all  links  are  of  equal  quality.  This  latter 
assumption  is  not  necessary  but  will  be  used  in  order  to  simplify  matters.  As 
mentioned  in  Paragraph  3.3. 7.4,  10  bits  are  sufficient  for  nodal  ranking 
information  and  7  bits  are  sufficient  for  specifying  the  number  of  nodes  between 
the  local  node  and  the  master  reference  node.  The  frequency  with  which  this 
information  needs  to  be  updated  depends  on  how  rapidly  the  local  node  can  utilize 
the  information.  It  is  not  believed  that  the  local  node  needs  this  information 
more  often  than  it  needs  to  update  its  local  loop.  Therefore,  let  us  assume  that 
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the  self-organizing  information  Is  updated  1.5  times  per  second.  Table  3.3.11.5-1 
shows  permanent  storage  locations  associated  with  the  self-organizing  feature. 
Figure  3.3.11.5-1  shows  a  flow  chart  of  the  operations  performed  by  the  timing 
subsystem  processor  to  implement  the  self-organizing  feature  using  the  data  of 
Table  3.3.11.5-1.  The  flow  chart  of  Figure  3.3.11.5-1  was  coded  in  the  8080A 
machine  language  and  may  be  excuted  by  the  8080A  microprocessor  using  the 
following  time  and  space: 

Time  -  (927  x  366n)  x  0.5  jjS  per  iteration 

Space  -  (227  +  3n)  bytes, 

where  n  is  the  number  of  neighbor  nodes.  For  n=10  this  requires  4593  /is/ 

Iteration  and  307  bytes  of  storage  space.  At  1.5  iterations  per  second  this 
represents  approximately  4.593  x  10-3  x  1.5  x  100%  =  0.69%  of  the 
microprocessor's  time. 

3.3.11.6  Summary 

Table  3.3.11.6-1  summarizes  the  processor  time  and  memory  space 
required  to  implement  each  of  the  nodal  timing  subsystem  features.  From  this 
table  it  is  evident  that  the  timing  subsystem  processor  poses  no  limitations  to 
implementing  any  of  the  features  if  the  assumptions  that  were  made  in  the  analysis 
are  realized.  Thus  the  only  real  impediment  to  implementing  any  set  of  the 
features  is  the  overhead  channel.  It  has  been  shown  in  Paragraph  3.3.7  that  the 
bit  rates  required  for  the  overhead  function  are  quite  modest.  On  the  other  hand, 
the  overhead  provisions  may  impact  the  permissible  multiplexing  structure.  This 
comes  mainly  in  the  form  of  potential  inflexibility  that  may  or  may  not  surface  at 
some  future  time.  However,  these  potential  inflexibilities  have  not  been 
discovered  during  the  course  of  this  study. 
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Table  3.3.11.5-1.  Permanent  Storage  Locations  Associated  With 
The  Self-Organizing  Feature 


RANK  OF  LOCAL  NODE  (SET  BY  LOCAL 
OPERATOR  OR  NETWORK  CONTROLLER) 

PATH  LENGTH  FROM  LOCAL  NODE  TO  LOCAL  NODE 

RANK  OF  NEIGHBOR  NO.  VS  MASTER  REFERENCE 

PATH  LENGTH  TO  MASTER  REFERENCE 

RANK  OF  NEIGHBOR  NO.  2'S  MASTER  REFERENCE 


PATH  LENGTH  TO  MASTER  REFERENCE 


RANK  OF  NEIGHBOR  NO.  n‘S  MASTER  REFERENCE 

PATH  LENGTH  TO  MASTER  REFERENCE 

POINTER  TO  PRESENT  ENTRY  WITH  HIGHEST 
REFERENCE  RANK  AND  SHORTEST  PATH 

POINTER  TO  PREVIOUS  ENTRY  WITH  HIGHEST 
REFERENCE  RANK  AND  SHORTEST  PATH 

POINTER  TO  ENTRY  USED  BY  LOCAL  NOOE 
TO  OBTAIN  REFERENCE 


COUNTER  FOR  NO.  OF  TIMES  LOCAL  NOOE 
DOES  NOT  REFERENCE  HIGHEST  RANK  AND 
SHORTEST  PATH 


10066  16 
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Table  3.3.11.6-1.  Summary  of  Processor  Time  and  Space  Required  For 

The  Timing  Subsystem  Features 

Feature 

Estimated 

X  Processor  Time 

Estimated  Storage 
Space  (bytes) 

Directed  Control 

0.768 

395 

Mutual  Control 
(10  neighbor  nodes) 

7.2 

594 

Double  Ended  (4  over  single  ended) 
Directed  (10  neighbors  below) 

1.47 

336 

Mutual 

(10  neighbors) 

3.345 

451 

Independence  of  Clock  Error 
Measurement  and  Correction 
(  4  over  double  ended) 

Directed 

Mutual 

0.177 

N/A 

71 

N/A 

Phase  Reference  Combining 
(10  neighbors) 

Directed  only 
(  A  over  directed) 

8.23 

1045 

Self-Organizing 
(10  neighbors) 

Directed  only 

0.69 

307 

2501A 


3.3.12  Interoperability 

3.3.12.1  Meaning  of  Interoperability 

Interoperability  refers  to  the  ability  of  the  digital  DCS  network, 
which  may  have  one  or  more  of  the  network  synchronization  subsystem  features,  to 
operate  with  other  communication  systems  such  as  the  military  tactical 
communications  network  and  comnercial  communications  and  data  networks.  The 
primary  emphasis  here  is  on  the  ability  to  coirmunicate  across  interface  points 
between  the  DCS  network  and  external  communications  networks.  Secondary 
considerations  are  concerned  with  the  transfer  and  utilization  of  frequency  and 
time  reference  signals  between  the  DCS  network  and  external  communications, 
navigation,  and  position  location  networks.  The  point  of  interface  between  the 
DCS  network  and  the  external  systems  may  occur  anywhere  from  the  highest  level  to 
the  lowest  level  of  the  DCS  network.  The  level  in  the  DCS  network  at  which  the 
interface  with  the  external  system  occurs  influences  the  interoperability 
charac ter istic. 

3.3.12.2  Feature  Enhancement  for  Interoperability 
3.3.12.2.1  Directed  Control 

The  major  contribution  that  the  directed  control  feature  can  provide 
for  the  DCS  network  synchronization  subsystem  in  enhancing  interoperability,  in  a 
communications  sense,  occurs  when  both  the  DCS  network  and  the  external  networks 
are  disciplined  at  their  highest  levels  by  signals  derived  from  UTC  (USNO)  or  UTC 
(NBS).  In  this  case  the  networks  can  interoperate  at  any  point  with  the  directed 
control  feature  intact  across  network  boundaries  just  as  nodes  within  the  DCS 
network  but  in  different  directed  control  chains  are  interoperable  with  each 
other.  Even  though  the  directed  control  feature  may  be  applied  to  the  DCS  network 
there  is  a  condition  when  communications  between  a  node  in  the  DCS  network  and  a 
node  in  an  external  network,  from  a  synchronization  standpoint,  is  the  same  as  it 
is  for  nodes  utilizing  independent  clocks.  This  condition  occurs  when  no 
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frequency  and  time  reference  signals  are  exchanged  and  utilized  between  the  two 
networks  and  no  common  reference  signals  such  as  UTC  (USNO)  or  UTC  (NBS)  are 
applied  to  both  the  DCS  and  external  networks.  However,  the  apparent  independent 
clock  on  the  DCS  side  of  the  interface  point  may  appear  to  be  a  high  quality  clock 
due  to  being  disciplined  by  means  of  the  directed  control  feature.  If 
synchronization  and  time  reference  signals  are  transferred  between  the  DCS  network 
and  an  external  system  to  be  utilized  in  disciplining  clocks  in  the  DCS  network 
which  utilizes  the  directed  control  feature  then  the  point  of  application  in  the 
DCS  network  should  be  at  the  highest  level  node  of  the  DCS  network.  In  this  case 
the  two  networks  combine  into  a  larger  network  with  the  integrity  of  the  directed 
control  feature  intact.  If  the  interface  point  of  the  DCS  network  is  not  at  the 
highest  level  node  then  the  directed  control  feature  cannot  be  maintained. 
Synchronizati on  and  time  reference  signals  can  be  transferred  from  any  point  of 
the  DCS  network  to  external  systems  without  affecting  the  integrity  of  the 
directed  control  feature  of  the  DCS  network.  In  fact,  if  this  is  done  with  more 
than  one  external  system  and  the  DCS  network  utilizes  the  directed  control 
feature,  then  the  interoperability  of  these  external  systems  among  themselves  will 
be  enhanced  because  their  time  and  frequency  reference  signals  will  be  correlated 
although  they  may  be  applied  from  different  points  of  the  DCS  network. 

3.3.12.2.2  Mutual  Control 

The  mutual  control  feature  does  not  rule  out  the  ability  of  the  DCS 
network  to  interoperate  with  external  networks.  As  for  directed  control,  there  is 
a  set  of  conditions  under  the  mutual  control  regimen  in  the  DCS  network  whereby 
communications  between  a  node  of  the  DCS  network  and  a  node  of  an  external  network 
are  the  same,  from  a  synchronization  standpoint,  as  it  is  for  nodes  utilizing 
independent  clocks.  This  set  of  conditions  occurs  when  mutual  control  without  a 
master  is  applied  to  the  DCS  network  and  no  exchange  and  utilization  of  reference 
information  is  made  between  the  DCS  and  external  networks.  However,  the  frequency 
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of  bit  slips  at  such  interface  points  may  be  decreased  due  to  improved  stability 
of  the  clock  on  the  DCS  side  of  the  interface  provided  by  mutual  control 
disciplining.  In  this  sense  the  mutual  control  feature  provides  enhancement  of 
the  interoperability  characteristic  over  that  of  independent  clocks.  If  frequency 
and  time  reference  information  is  exchanged  between  communicati ng  nodes  in  the  DCS 
network  and  an  external  network  and  both  nodes  utilize  this  information  to 
discipline  their  respective  nodal  clocks,  then  the  mutual  control  feature  may  or 
may  not  be  maintained  across  the  interface  in  the  sense  that  communication 
buffers  may  or  may  not  overflow  depending  on  the  tightness  of  coupling  of  the  two 
networks  through  this  and  other  similar  points  of  interface.  If  the  networks  are 
loosely  coupled  then  the  expectation  for  communication  buffer  slips  at  the 
interface  point  is  higher. 

3.3.12.2.3  Double  Ended 

The  double  ended  feature  takes  out  link  delay  variations.  Thus  if  the 
DCS  network  synchronization  subsystem  utilizes  this  feature  interoperability  with 
other  communications  networks  will  be  enhanced,  especially  if  the  other  network 
utilizes  the  double  ended  feature. 

3.3.12.2.4  Independence  of  Clock  Error  Measurement  and  Correction 

This  feature  removes  clock  perturbations  from  each  timing  chain  and 
consequently  provides  for  each  clock  to  be  effectively  controlled  by  the  ultimate 
master.  Some  contribution  to  interoperability  of  the  DCS  network  with  external 
communications  network  can  be  obtained  with  this  feature  because  the  transistory 
fluctuations  caused  by  intervening  clocks  are  removed  and  hence  nodes  at  the  far 
ends  of  the  network  remain  tightly  coupled  to  the  network  master  phase.  This 
allows  these  tail  end  nodes  to  more  readily  communicate  with  an  external  network. 

3.3.12.2.5  Self-Organizing 

The  self-organizi ng  feature  provides  for  quick  selection  of  new 
reference  sources  throughout  the  DCS  network  after  a  failure  occurs.  Thus  nodes 
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should  have  a  reference  for  larger  percentages  of  time  if  the  self-organizi  ng 
feature  is  used.  This  will  enhance  interoperability  with  external  networks 
because  each  node  of  the  DCS  network  can  on  the  average  be  expected  to  be  held 
closer  to  the  network  frequency. 

3.3.12.2.6  Overhead  Channel 

To  the  extent  that  the  overhead  channel  is  necessary  to  the 
implementation,  of  the  self-organizing,  double  ended,  and  independence  of  error 
detection  and  correction  features,  these  features  enhance  interoperability,  the 
overhead  channel  enhances  interoperability. 

3.3.12.2.7  Phase  Reference  Combining 

Phase  reference  combining  is  a  technique  for  reducing  measurement 
errors  in  the  dissemination  of  reference  information  to  the  various  nodes  of  the 
network.  It  also  provides  a  means  for  monitoring  the  condition  of  the  various 
reference  signals  arriving  at  each  node.  The  net  result  of  this  feature  is  that 
each  node's  phase  may  be  held  closer  to  the  phase  of  the  network's  master  clock. 

In  this  sense,  interoperability  with  external  networks  is  improved. 

3.3.12.3  Synerqisms  of  the  Features 

Interoperability  is  one  characteristic  for  which  maximum  enhancement  is 
obtained  when  several  of  the  network  synchronization  subsystem  features  and 
Federal  Standard  1002  are  simultaneously  incorporated.  Federal  Standard  1002  and 
one  of  the  disciplining  features  (directed  control  or  mutual  control)  provides  the 
basis  on  which  to  build  the  collection  of  features  to  produce  maximum 
i nteroperabi lity  through  symbiont  relations  between  these  additional  features  and 
the  basis  features.  The  additional  features  include  self-organizing, 
double-ended,  independence  of  clock  error  measurement  and  correction  and  phase 
reference  combining. 
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In  sumnary  it  is  emphasized  that  there  are  primary  features  and 
secondary  features  which  lead  to  interoperability  between  the  DCS  network  and 
external  communications  networks.  The  primary  features  are  the  network 
disciplining  features  (directed  or  mutual  control)  with  a  contnon  reference  such  as 
UTC  (USNO)  or  UTC  (NBS).  It  is  necessary  that  these  primary  features  be  applied 
to  both  the  DCS  network  and  the  external  network.  The  interoperability 
characteristic  can  then  be  further  improved  by  provision  of  those  secondary 
features  that  are  applicable  to  the  basic  disciplining  technique.  The  primary 
features  ensure  that  the  long  term  average  frequency  of  the  DCS  network  and 
exter  al  networks  is  the  same.  The  secondary  features  all  help  to  reduce  short 
term  perturbations  that  may  occur  anywhere  in  the  network  to  which  they  are 
appl i ed. 

3.3.13  Automatic  Selection  of  New  Master 

This  characteristic  is  provided  by  the  self-organizing  feature  which  is 
normally  thought  of  as  being  used  in  conjunction  with  the  directed  control 
feature.  In  a  strict  sense  the  mutual  control  feature  does  not  contribute  to  this 
characteristic.  However,  under  a  somewhat  more  relaxed  set  of  definitions  the 
mutual  control  scheme  obviates  the  need  for  such  a  characteristic.  Thus  the 
question  is  mute  when  applied  to  a  timing/synchronization  subsystem  which  utilizes 
mutual  control.  On  the  other  hand,  it  is  possible  to  relax  the  definition  of 
mutual  control  so  as  to  allow  for  some  clock  or  clocks,  either  a  part  of  the 
network  or  not  in  a  communications  sense,  to  influence  the  phase  and  frequency  of 
other  clocks  of  the  network  while  themselves  not  being  reciprocally  influenced  by 
clocks  of  the  network.  If  there  is  exactly  one  such  clock  in  this  category  then 
some  people  call  this  a  mutual  control  scheme  with  a  master.  Under  such  a 
disciplining  scheme  the  self-organizing  feature  may  also  be  applied.  Thus  in 
either  case  (directed  control  or  mutual  control  with  a  master)  the  self-organizing 
feature  must  be  applied  in  order  to  satisfy  this  characteristic.  See  also  the 
discussion  in  Section  2. 1.4.1. 


3-170 


3.4 


SUMMARY 


An  assessment  has  been  made  of  the  value  of  timing/synchronization 
subsystem  features  to  the  provision  of  a  given  set  of  13  desirable  characteristics. 
This  assessment  effort  was  oriented  toward  performance  evaluation  of  the  features 
and  combinations  thereof  and  made  no  attempt  to  impose  any  value  judgnents  on  the 
desirable  characteristics.  Value  judgments  can  be  most  readily  made  by 
appropriate  government  agencies  who  have  a  broader  knowledge  of  future 
requirements  and  uses  of  the  DCS  digital  network.  Once  such  value  judgments  have 
been  made  the  results  of  the  present  assessment  should  provide  valuable 
information  in  the  selection  of  those  timing/synchronization  subsystem  features 
that  are  most  appropriate  for  the  DCS  network. 

An  underlying  assumption  in  this  assessment  was  that  implementations  of 
the  nodal  synchronizer  would  involve  some  form  of  microcomputer.  The  control 
mechanism  at  each  node  was  a  phase-locked  loop  utilizing  an  extended  range  linear 
phase  detector.  A  total  of  11  features  were  utilized  in  the  assessment.  Seven  of 
the  features  were  considered  to  be  primary  features  and  four  were  considered  to  be 
secondary  features.  A  small  subset  (16)  of  the  feature  combinations  was  chosen 
for  use  in  the  evaluation.  This  subset  was  arrived  at  through  analysis  on  the 
basis  of  which  combinations  are  logically  permissible  and  most  desirable.  Six  of 
the  desirable  characteristics  were  of  such  complexity  as  to  require  a  digital 
computer  simulation  model  to  provide  narrow  information  which  would  aid  in  the 
evaluation.  The  remaining  seven  desirable  characteristics  were  sufficiently 
simple  that  evaluation  could  be  accomplished  through  analysis  that  incorporated 
broad  information  gleaned  from  the  simulation  effort  and  other  sources. 

Table  3.4  is  a  sunmary  of  the  assessment  results.  In  Table  3.4  all 
entries  with  the  exception  of  overhead  requirements,  cost  effectiveness,  precise 


<D 


ctn  c\j 

*-«  rH 


40covor^covOKraococvj 


•H  CO  CO  rH  rH  CVJ  rH  rH 


»  o*Tvocvjcor-HCOa>r^vovo*i-Pr>cvj 

I  rH  rH  rH  rH  rH  rH 


CO  CO  Kf  CO  CO  Kf  CO^TCVJrHrHCOCVJ.— 4*— 4 


*— I  *-H  00  ID  VO  ^3"  r-HCOC\Jf-Hf-Hr-Hr-Hr-H 


co  cvj  vr>  vo  co  o  ^  co 


vo  vo  r-H  cvj  *-h 


o  o  o 

^  ?  5? 


V. 

<D 

4-> 

/. 

u 

«  y 

r  -5 

40 

-C  / 

CD 

«->/ 

La- 

LiJ 

UJ 

VO 

LaJ 

CO 

1 

to 

1 

2 

1 

2 

UJ 

2 

LiJ 

zd 

LaJ 

ZD 

i 

1 

t 

X 

i 

O 

X 

i 

• 

VO 

VO 

o  o  o 

CO 

I  I  I 


o 
to  a. 

i  i 

o  o 

2  2 

LjJ  UJ 
O  C_> 

*— •  »—4 

I  I 


t  «  - 


CO 

8 

9 

1 

2 

ZD 

1 

Q 

1 

Q 

1 

to 

1 

o 

» 

9 

9 

u 

«TJ 

•r 

4-> 

iO 

L_ 

8 

CVJ 

LaJ 

X 

CVJ 

CVJ 

CVJ 

1 

CVJ 

CVJ 

CVJ 

JC. 

C 

<D 

1 

• 

1 

1 

1 

1 

i 

1 

1 

a> 

JO 

r— 

O 

X 

8 

O 

* 

o 

o 

8 

o 

o 

8 

o 

o 

8 

H-» 

o 

o 

1 

40 

E 

z 

a. 

z 

tn 

• 

• 

• 

CD 

• 

r-H 

CVJ 

• 

CO 

VO 

vo 

00 

»H 

rH 

r-H 

rH 

rH 

rH 

rH 

TO  <T3 

C  O 
0>  •*- 
Q.  TO 
Q  Q.  C 
<  **“ 
X 

r-  <D  VO 
TO  d> 

C  CO  .C 

a>  cn 

CL  *3 
Q.  C  O 
O  U 

$  5  ^ 

CO  C  rH 

•r— 

*  tf-  t/> 
C  OJ  c 

O  TO  E 


4-»  O  «— 

—  4-»  O 

C  U 

•r-  C7> 

««-  C  U 

a»  o  • 

TO  TO  4-  -O 

W  <D 

O  O  0>  U 

P  u  *-  L 

O  JO  <U 

w  <o  V  • 

C  -MO) 

•r-  »—  U 

TO  3  0>  Q.  , 

U  *-  -C 

O  £  4-»  VC 

<->  i-  l- 

o  <o  a>  <y 

r  tj  Z 

^  —  e 

r—  >»  cn  13 

3  •—  C  c 


cn 

•  4 

LO 

CVJ 

CVJ 

o% 

§ 

Vj 

<Ts 

<T> 

(T» 

cr> 

CO 

*3- 

VO 

O 

o 

o 

LO 

CO 

LO 

IT) 

lo 

U"> 

CO 

o 

r—i 

00 

00 

o 

tH 

*— H 

rH 

t  < 

r— H 

rH 

CVJ 

O 

O 

O 

o 

O 

to 

o 

CVJ 

LO 

VO 

CO 

r— H 

*3“ 

CVJ 

CVJ 

CVJ 

CVJ 

CVJ 

CVJ 

to 

VO 

VO 

^3- 

O' 

CO 

O 

r>- 

CVJ 

o 

CVJ 

o 

i  4 

CVJ 

CO 

rH 

TO 

8 

TO 

O 

O 

TO 

O 

O 

TO 

O 

O 

TO 

O 

O 

TO 

O 

O 

TO 

8 

TO 

O 

O 

TO 

8 

TO 

O 

O 

TO 

8 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

compared  to  the  usual  amount  of  overhead  space  occupied  by  frame  sync  patterns. 
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time  availability,  and  selection  of  new  master  represent  values  that  are  relative 
to  other  entries  in  the  same  column.  In  the  column  titled  "Clock  errors  do  not 
harmfully  propagate  to  other  nodes",  one  clock  error  of  10"^  p-p  was  judged  to 
be  potentially  harmful  to  some  node  of  the  network  under  each  set  of  feature 
combinations  except  the  ones  indicated.  In  the  "precise  time  availability"  column 
the  YES  entries  do  not  mean  that  precise  time  is  automatically  provided  by  this 
combination  of  features  but  rather  indicate  that  the  essential  ingredients  are 
provided  to  support  an  add  on  unit  to  the  nodal  synchronizer  which  gives  precise 
time.  In  the  column  titled  "selects  new  master"  the  mutual  control  systems 
utilizing  a  master  node  (5,6,  and  10)  had  no  provisions  for  automatic  selection  of 
a  new  master  and  thus  were  marked  with  NO.  However,  a  real  implementation  of  such 
a  system  might  requi re  provisions  for  automatic  selection  of  a  new  master. 

In  addition  to  those  specific  conclusions  made  in  each  individual 
section,  the  following  overall  conclusions  have  been  made: 

OVERALL  CONCLUSIONS 

•  All  features  tend  to  do  the  things  which  they  are  logically 

designed  to  do.  They  all  provide  contributions  to  the  attainment 
of  subsets  of  the  desirable  characteristics. 

Directed  and  mutual  control  tend  to  provide  for  a  synchronized 
network  with  long  term  frequency  averages  at  each  node  the  same  as 
or  very  close  to  the  network  frequency.  Additionally,  directed 
provides  for  long  term  zero  average  phase  error  and  contains 
disturbances  in  branch  in  which  they  occur  and  only  propagate  them 
downward. 

Double-ended  removes  path  delay  variations. 

Smoothing  removes  undesirable  large  frequency  spikes  due  to  step 
changes  in  reference  phase. 
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Master  in  mutual  system  provides  definite  network  frequency. 

Unequal  weighting  in  mutual  system  tends  to  improve  short  term 
accuracy  but  can  cause  larger  transients  with  link  and  nodal 
failures. 

ICEM&C  removes  disturbances  due  to  independent  clock  errors  in  a 
branch  of  nodes. 

Phase  reference  combining  is  effective  in  combatting  measurement 
jitter  and  also  decreases  expected  percentage  of  time  that  nodes 
may  be  without  a  reference  during  periods  of  stress. 

•  The  simulations  were  performed  with  limited  network  size  and 
connectivity  and  for  very  limited  run  time.  With  larger  networks 
and  connectivity  and  much  longer  run  times  the  expected  benefits 
from  the  additional  features  will  be  accentuated. 

•  Provisions  for  additional  features  (over  and  above  mutual  or 
directed  control)  does  not  seem  excessive  in  that  overhead  data 
requirements  are  quite  small  and  processor  time  and  storage  space 
is  small  in  comparison  with  the  capabilities  of  presently  available 
microcomputers. 

•  The  most  striking  of  the  additional  features  is  the  double-ended 
reference  links. 

•  Although,  according  to  the  definition  of  harmful  transient,  most  of 
the  disturbances  due  to  clock  errors  or  path  delay  variations  and 
dropouts  experienced  in  the  simulation  were  judged  to  be 
non-harmful  these  events  were  mostly  isolated  and  their  amplitudes 
were  chosen  to  represent  typical  events.  In  a  real  stressed 
environment  it  is  possible  that  several  such  events  could  occur 
closely  enough  together  in  time  and  at  the  correct  points  in  the 
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network  to  be  harmful.  The  tabular  surmary  indicates  the 
combinations  of  features  least  vulnerable  to  such  threats. 

•  Precise  time  can  be  provided  as  an  add-on  feature  to  any  scheme  of 
control  that  has  a  master  and  utilizes  double-ended  links.  The 
add-on  does  not  affect  the  manner  in  which  the  nodal  synchronizer 
controls  the  local  clock's  phase  and  frequency.  Additional 
features  may  be  used  to  improve  the  accuracy  of  disseminated  time. 

•  Without  a  master  the  network  frequency  of  the  mutual  control  system 
may  wander  which  makes  interoperation  difficult.  Provision  of  a 
master  would  then  dilute  "claimed"  survivability  of  this  method  of 
control.  Overhead  is  also  required  to  automatically  select 
alternate  masters. 

•  A  disturbance  occurring  anywhere  in  the  mutual  control  network 
propagates  to  all  nodes  of  the  network. 

•  In  order  to  ensure  network  stability  in  a  mutual  control  system 
limitations  are  placed  on  the  type  of  nodal  loop.  The 
disadvantages  are  as  follows: 

1.  Type  1  loop  tracks  constant  frequency  offset  with  non-zero 
phase  error. 

2.  Type  1  loop  tracks  constantly  drifting  clock  with  non-zero 
frequency  error  and  constantly  increasing  phase  error. 

3.  The  above  two  characteristics  tend  to  degrade  the  short  term 
accuracy  of  the  network. 

4.  Characteristic  2.  above  indicates  need  for  special  provisions 
for  drifting  clocks,  size  buffer  for  lifetime  operation, 
periodic  adjustment  of  natural  frequency,  or  other  means. 
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•  Error  history  at  each  node  in  mutual  control  system  is  a  complex 
function  involving  network  topology,  reference  weightings,  stress 
events,  and  individual  clock  performance  at  all  other  nodes  of  the 
network.  This  makes  it  more  difficult  to  devise  a  control  strategy 
during  intervals  when  a  reference  is  not  available.  Thus 
survivability  is  decreased.  This  complex  history  also  lessens  the 
utility  of  monitored  parameters  at  each  node. 
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4.0  ANALYSIS  OF  COST  VERSUS  FEATURES 

The  hardware  and  software  implementations  of  the  Timing  Subsystem 
features  described  in  Paragraph  2.2  are  addressed  in  this  section.  Eleven  Timing 
Subsystem  configurations  possessing  these  features  in  all  practical  combinations 
were  designed  and  their  life-cycle  costs  were  determined.  The  costs  of  these 
configurations  may  be  used  in  conjunction  with  the  performance  evaluations  of 
Section  3.0  to  evaluate  performance  versus  cost  trade-offs. 

4.1  Generalized  Timing  Subsystem 

The  Timing  Subsystem  consists  of  all  components  not  included  in  the 
radios,  modems,  multiplexers,  data  terminals,  etc.,  which  are  necessary  to  provide 
timing  for  and  to  synchronize  data  transfer  between  said  equipments.  The  broad 
categories  of  these  components  are:  local  reference,  buffers,  timing  generation 
and  distribution,  and  optional  disciplining  circuitry. 

4.1.1  Conceptual  Design 

Figure  4. 1.1-1  shows  the  assumed  multiplexer  hierarchy  used  for  the 
design  analysis.  Some  liberties  had  to  be  taken  in  the  use  of  these  equipments  in 
order  to  implement  the  nodal  synchronization  function.  It  should  be  noted  that  a 
new  generation  of  equipments  may  be  desirable  for  use  in  the  future  synchronous 
DCS.  Since  it  would  be  difficult  to  estimate  the  cost  impact  of  redesigning  these 
equipments,  the  liberties  taken  with  their  use  were  kept  to  an  absolute  minimum. 
Hence,  the  Timing  Subsystem  was  designed  around  the  present  equipment 
specifications  as  much  as  practical,  and  the  resulting  Timing  Subsystem  costs  may 
be  somewhat  higher  as  a  result. 

Figure  4. 1.1-2  is  a  generalized  block  diagram  of  the  Timing  Subsystem. 
The  system  depicted  here  contains  the  maximum  hardware  and  can  implement  all  of 
the  optional  features.  The  detailed  implementation  of  each  of  these  components  is 
described  in  Paragraph  4.2. 
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Figure  4. 1.1-1.  Transmit  and  Receiver  Configurations 
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Figure  4. 1.1-2.  Timing  Subsystem 
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The  elastic  buffer  absorbs  rate  variations  between  the  received  timing 
and  the  local  clock.  An  elastic  buffer  is  placed  between  each  of  the  two  channels 
of  the  digital  radio  and  the  two  corresponding  TD-1193  Demultiplexers.  (Only  one 
is  shown  for  brevity.)  These  buffers  could  be  placed  lower  in  the  demultiplexer 
hierarchy,  resulting  in  a  larger  number  of  smaller,  slower  buffers.  An  analysis 
of  optimum  buffer  placement  was  not  included  in  this  effort;  they  are  assumed  to 
be  between  the  radio  and  highest-level  demultiplexer  for  ease  of  analysis. 

Approaches  which  discipline  a  clock  based  on  time  of  arrival  of  link 
timing  signals  require  a  sync  extractor  and  phase  detector.  The  sync  extractor 
searches  the  mission  bit  stream  for  the  TD-1193  sync  patterns.  The  phase  detector 
measures  the  time  differences  between  the  predicted  and  actual  arrival  times  of 
these  sync  patterns.  The  time  differences  (phase  errors)  are  inputs  to  the 
phase-locked  loop  which  serves  as  the  local  time  reference. 

The  processor  inputs  these  phase  errors  and  calculates  the  difference 
equations  which  implement  the  digital  loop  filter.  The  output  samples  from  this 
filter  discipline  the  nodal  clock.  They  may  control  a  VCXO  (voltage  controlled 
crystal  oscillator)  via  a  D/A  (digital-to-analog)  converter,  or  may  discipline  an 
atomic  standard  by  shifting  its  phase  via  an  outboard  phase  shifter.  The  nodal 
clock  drives  the  frequency  synthesizers  which  produce  all  phase-related  timing 
required  by  the  multiplexers,  radios,  modems,  etc.,  and  all  demultiplexers  which 
are  lower  in  the  hierarchy  than  the  elastic  buffers. 

Timing  approaches  which  employ  overhead  information  require  an  overhead 
interface  which  allows  information  to  be  exchanged  from  processors  on  opposite 
ends  of  the  link.  It  is  assumed  that  this  information  is  transmitted  through  a 
low-speed  channel  (300  b/s)  of  the  orderwire  multiplexer  (LSTDM). 

4.1.2  Implementation  of  Features 

Each  of  the  features  described  in  Paragraph  2.2  were  analyzed  to 
determine  their  hardware  and  software  requirements. 
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4. 1.2.1 


Directed  Control 


Directed  control  requires  at  least  one  sync  extractor  and  phase 
detector  in  the  system.  For  single-ended  systems,  the  input  to  this  circuitry  is 
multiplexed  so  that  it  may  connect  to  any  one  of  the  incoming  links.  This  feature 
also  requires  a  loop  filter  (processor)  and  means  for  controlling  the  local  clock. 

4. 1.2.2  Double-Endedness 

Double-endedness  under  Directed  Control  requires  that  the  phase  error 
measurement  be  made  on  each  end  of  the  link  and  that  the  master  transmit  its 
measured  phase  error  to  the  slave  via  the  overhead  channel.  The  local  node  must 
contain  a  sync  extractor  and  phase  detector  for  the  link  to  which  it  is  slaved, 
and  a  sync  extractor  and  phase  detector  for  each  neighboring  node  which  is  slaved 
to  the  local  node.  Further,  the  local  node  must  contain  an  overhead  receiver  for 
the  link  to  which  it  is  slaved,  and  an  overhead  transmitter  for  each  neighboring 
node  which  is  slaved  to  the  local  node. 

Double-Endedness  under  Mutual  Control  requires  an  overhead  transmitter 
and  receiver  for  each  connecting  link. 

This  feature  also  requires  an  additional  computation  (subtraction  and 
division  by  2)  to  be  performed  by  the  processor.  Under  Mutual  Control,  this 
computation  is  required  for  each  connecting  link. 

4. 1.2. 3  Independence  of  Clock  Error  Measurement  and  Correction 

This  feature  is  only  implemented  after  implementing  both  Directed 
Control  and  Double-Endedness.  No  additional  hardware  is  necessary.  The  Master 
transmits  its  measured-but-uncorrected  error  (relative  to  the  ultimate  master)  to 
the  slave  along  with  the  measured  phase  error  used  for  double-endedness.  The 
slave  subtracts  this  number  from  the  result  of  its  double-endedness  calculation. 
Thus,  only  a  minute  amount  of  additional  software  is  required.  It  should  be  noted 
that  the  Master  node  could  combine  these  two  numbers  and  transmit  them  as  one 
number,  thus  requiring  no  additional  bandwidth. 
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4.1.?. 4  Phase  Reference  Combining 

The  Phase  Reference  Combining  feature  is  implemented  only  after 
implementing  Directed  Control,  Double-Endedness,  and  Independence  of  Measurement 
and  Correction.  The  local  node  derives  timing  from  all  nodes  not  lower  in  the 
hierarchy,  and  therefore  must  contain  a  sync  Extractor  and  phase  Detector  for  each 
of  these  links.  Further,  all  nodes  not  higher  in  the  hierarchy  derive  their 
timing  in  part  from  the  local  node.  Since  a  double-end  measurement  is  used,  a 
sync  extractor  and  phase  detector  are  required  on  these  links.  Thus,  a  sync 
extractor  and  phase  detector  are  required  for  each  link. 

The  usual  overhead  transmission  required  for  double-endedness,  plus  the 
measured-but-uncorrected  phase  error  and  a  variance  estimate,  must  be  received 
from  all  nodes  not  lower  in  the  hierarchy.  The  same  three  information  types  must 
be  transmitted  to  all  nodes  not  higher  in  the  hierarchy.  Thus  overhead  receivers 
are  required  on  all  links  connecting  to  nodes  not  lower  in  the  hierarchy,  and 
overhead  transmitters  are  required  for  all  links  connecting  to  nodes  not  higher  in 
the  hierarchy.  For  example,  if  the  local  node  connected  to  three  nodes  higher  in 
the  hierarchy,  three  nodes  of  the  same  level,  and  four  nodes  lower  in  the 
hierarchy,  it  would  require  six  receivers,  and  seven  transmitters. 

Additional  software  is  required  to  implement  Phase  Reference 
Combining.  Paragraph  3.3.11.4  discusses  the  software  requirements. 

Rule  11,  Paragraph  ?.2.4,  requires  the  three  data  types  to  be  exchanged 
in  both  directions  on  all  links.  This  additional  information  is  used  for 
diagnostic  purposes  only,  and  is  not  required  for  implementing  Phase  Reference 
Combining.  If  Rule  11  is  implemented,  then  overhead  receivers  and  transmitters 
are  required  on  all  links,  and  additional  software  is  required. 

4. 1.2. 5  Self-Organization 

The  self-organization  feature  requires  overhead  information  to  be 
passed  in  both  directions  over  all  links,  i.e.,  the  processor  at  the  local  node 
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has  full  duplex  communication  with  the  processors  at  all  neighboring  nodes.  Thus, 
an  overhead  receiver  and  transmitter  is  required  on  each  link. 

Additional  bandwidth  and  processing  are  also  required.  For  a  Directed 
Control  System  not  containing  Phase  Reference  Combining,  each  node  selects  the 
best  link  to  serve  as  its  reference  based  on  three  data  types:  nodal  rank, 
distance  from  master,  and  link  demerit.  This  scheme  adheres  to  rules  similar 
to  those  of  Darwin  and  Prim215.  If  Phase  Reference  Combining  is  included,  the 
self-organization  feature  is  implemented  differently.  The  local  node  does  not 

have  to  select  the  best  link  for  its  reference  since  it  is  always  deriving  timing 

from  all  neighbors  not  lower  in  the  hierarchy.  However,  it  must  know  at  all  times 
which  of  its  neighbors  are  higher,  lower,  or  on  an  equal  level  within  the 
hierarchy.  Two  information  types,  INFO  1  and  INFO  2,  are  employed  for  this 
determination. 

4. 1.2. 6  Mutual  Control 

Mutual  Control  requires  a  sync  extractor  and  phase  detector  on  each 
incoming  link.  In  addition  to  the  loop  filter  software  (which  is  of  equal 
complexity  with  that  required  for  Directed  Control)  the  Mutual  Control  feature 
requires  a  weighted  average  of  the  phase  errors  derived  from  the  individual  links. 
4.2  Hardware/Software  Realizations 

The  various  components  shown  in  Figure  4. 1.1-2  were  designed  in  enough 
detail  to  determine  a  parts  list  plus  space  and  power  requirements.  A  standard 
packaging  scheme  widely  used  at  Harris  ESD  was  selected  as  the  basis  for  design. 

No  claim  is  made  that  this  packaging  scheme  is  optimal  for  this  particular 
application.  This  scheme  employs  4.5  inches  x  5.25  inches  double-sided  printed 
circuit  (PC)  cards  which  plug  into  23-slot  motherboards  via  80-pin  connectors. 
Figure  4.2-1  is  a  photograph  of  an  8080  processor  card  with  associated  memory  and 
I/O  cards  which  illustrate  the  type  of  PC  cards  assumed  in  the  design.  This 
processor  card  family  was  developed  on  an  Internal  R&D  program  at  Harris  ESD  and 
was  used  in  the  design  of  the  subsystem. 
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Figure  4.2-1.  Processor  Family 


Figure  4.2-2  depicts  the  drawer  design  used.  This  drawer  was  assumed 
to  be  modified  to  meet  the  EMI  requirements  which  have  been  specified  for  such 
equipments  as  the  digital  radio  and  various  multiplexers.  Two  motherboards  (each 
holding  up  to  23  cards)  can  be  mounted  horizontally  in  the  front  of  the  drawer, 
with  enough  room  in  the  rear  for  power  supplies. 

Two  drawers  are  necessary  to  house  the  circuitry  for  most 
configurations.  It  was  deemed  desirable  to  separate  the  components  which  would  be 
common  to  all  approaches  from  those  which  would  be  configuration  dependent. 
Components  common  to  all  approaches  are  the  frequency  synthesizers,  distribution 
amplifiers,  and  elastic  buffers.  These  components  are  packaged  in  a  single  drawer 
herein  called  the  Basic  Drawer,  which  is  constant  across  all  conf igurations.  This 
was  a  fortunate  choice  since  a  frequency  distribution  system  currently  being 
procured  for  DCS  II  may  also  be  usable  in  the  future  DCS,  and  buffers  are 
currently  being  designed  into  the  multiplexers.  This  drawer  was  included  in  the 
cost  figures  to  keep  the  cost  of  the  various  features  in  perspective;  its  cost  may 
be  easily  factored  out  to  more  closely  compare  the  costs  of  the  various  approaches 
used  in  the  second  drawer  (herein  called  the  Disciplining  Drawer). 

With  the  exception  of  the  frequency  synthesizers,  which  contain  some 
ECL  (Emiter-Coupled  Logic),  the  designs  incorporate  the  more  economical  and  less 
power  consuming  low-power  Schottky  TTL  (Transistor-Transistor  Logic).  Since 
differences  in  delay  in  the  systems  at  opposite  ends  of  the  link  contribute  to 
path  delay  asymmetry,  critical  paths  such  as  in  phase  detectors  are  implemented 
with  high-speed  Schottky  which  has  a  lower  absolute  gate  delay  variation. 

Both  drawers  contain  power  supplies  and  motherboard  wiring  sufficient 
to  support  the  PC  cards  required  for  seven  terminating  links  (the  assumed 
maximum).  It  was  assumed,  however,  that  the  average  node  would  only  interconnect 
with  four  other  nodes.  Thus  the  costing  was  accomplished  assuming  a  main  frame 
capable  of  supporting  seven  links,  but  populated  with  PC  cards  to  support  four 
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links.  The  following  paragraphs  describe  the  Implementations  of  the  Individual 
components. 

4.2.1  Phase  Detector 

Figure  4. 2. 1-1  depicts  the  sync  extractor  and  phase  detector 
circuitry.  The  clock  output  from  the  digital  radio  and  the  local  10  MHz  reference 
are  divided  down  to  a  common  8  kHz  where  the  phase  comparisons  are  made. 

Phase  difference  is  measured  by  counting  cycles  of  the  10  MHz  reference 
between  the  rising  edge  of  the  8  kHz  wave  derived  from  the  link  timing,  and  the 
falling  edge  of  the  8  kHz  wave  derived  from  the  local  10  MHz  reference  as  shown  by 
the  diagonal  arrows  in  Figure  4. 2. 1-2.  Several  measurements  are  averaged  to 
obtain  good  resolution. 

The  detection  of  synchronization  patterns  occurring  at  some  submultiple 
of  8  kHz  (depending  on  the  selected  output  rate  of  the  TD-1193)  results  in  pulses 
from  the  sync  detector  synchronizing  the  countdown  chain  to  the  received  framing. 

A  countdown  chain  from  the  local  reference  controls  the  time  of  departure  of  the 
local  TD-1193's  frames  by  synchronizing  both  the  TD-1193  and  the  transmit  portion 
of  the  digital  radio  to  the  local  clock. 

Accurate  phase  measurements  are  obtained  by  averaging  the  counts 
obtained  in  several  successive  measurements.  Results  from  Section  3.0  indicate 
that  a  phase  measurement  needs  to  be  read  out  no  more  often  than  1.5  times  per 
second,  or  every  0.667  ms.  This  would  allow  in  excess  of  5,000  successive 
measurements  to  be  averaged.  Averaging  is  simply  accomplished  by  allowing  the 
counts  to  accumulate  in  the  accumulator  counter  until  readout  time,  and  dividing 
by  the  appropriate  constant.  At  zero  phase  error,  an  average  of  625  counts  will 
be  accumulated  for  each  measurement.  If  the  interval  counter  is  configured  to 
count  5,243  such  measurements,  then  3,276,875  counts  would  be  accumulated 
corresponding  to  zero  phase  error.  If  the  first  stage  of  the  accumulating  counter 
is  a  divide  by  100  and  the  remainder  a  16-bit  binary  counter,  this  16-bit  counter 
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would  contain  32,  /68  counts.  Complementing  the  MSB  would  yield  a  16-bit  two's 


complement  number  for  the  loop  filter.  This  phase  error  would  have  a  granularity 
of  62.5  n sec/ 32. 768  =  approximately  2  nsec.  This  represents  only  an  example  of 
several  possible  schemes. 

If  the  two  8  kHz  waves  are  perfectly  stationary  to  each  other  (or 
within  100  ns)  the  same  count  would  be  obtained  on  each  measurement.  Thus  the 
average  of  these  measurements  would  still  be  no  more  accurate  than  100  ns.  A  way 
to  improve  the  situation  is  to  purposely  jitter  the  link  clock  with  small 
constant-mean  delay  steps,  using  the  progranmable  delay  shown  in  Figure  4. 2. 1-1. 
Suppose  this  device  can  insert  delays  of  0  to  100  ns  in  10  ns  intervals. 

Advancing  this  device  10  ns  after  each  measurement  will  result  in  determination  of 
the  8  kHz  waves  leading  edge  with  a  10-ns  granularity.  Sequencing  the  delay 
cyclicly  from  0  to  100  ns  results  in  an  average  delay  of  50  nsec  over  10 
measurements  and  an  average  count  over  the  10  measurements  which  is  indicative  of 
the  location  of  the  rising  edge  of  the  8  kHz  wave  to  within  10  ns.  Smaller  delay 
variations  (random  ones)  will  be  generously  contributed  by  the  link.  This  results 
in  the  determi nation  of  the  average  phase  error  over  the  averaging  interval  to 
within  the  2  ns  granularity  of  the  counter.  Identical  progranmable  delays  on 
opposite  ends  of  the  link  preserve  symmetry.  This  2  ns  granularity  in  measurement 
is  better  than  the  asymmetry  of  the  link  and  the  associated  equipments  and  is  thus 
more  than  an  adequate  measurement  granularity.  This  phase  detector  has  an 
extended  range  of  +62.5 /zs.  The  sync  extractor  requires  one  PC  card,  and  the 
phase  detector  requires  two  cards. 

The  averaging  technique  described  above  using  a  10  MHz  clock  is 
believed  to  be  less  expensive  than  employing  a  higher  frequency  clock  to  achieve 
better  resolution.  Clock  frequencies  in  excess  of  10  MHz  impose  severe  design 
rules  on  the  logic  design.  Above  20  MHz,  a  transition  to  the  faster  ECL  family 
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with  attendant  proliferation  of  IC  counts  and  power  requirements  is  mandatory. 
Conmerci ally-produced  time  interval  meters  are  also  available  from  such 
manufacturers  as  Hewlett  Packard.  These  instruments  employ  an  internal  clock,  and 
obtain  their  resolution  by  averaging  several  measurements.  Given  that  some 
components  must  be  designed  for  the  timing  subsystem,  necessitating  an  enclosure 
and  power  supplies,  it  appears  most  economical  to  include  the  custom  designed 
counter  in  the  system  as  opposed  to  purchasing  an  expensive  general-purpose 
counter. 

4.2.2  Loop  Filter 

In  order  to  implement  a  filter  with  a  time  constant  on  the  order  of 
several  days,  a  digital  (as  opposed  to  analog)  filter  is  a  necessity.  An  8080 
microprocessor  was  chosen  to  implement  this  filter.  The  processor  card  shown  in 
Figure  4.2-1  is  a  self-contained  computer  including  IK  bytes  of  Programmable 
Read-Only  Memory  (PROM)  and  IK  bytes  of  Random  Access  Memory  (RAM).  This  single 
card  is  sufficient  to  perform  the  loop  filter  function.  An  analysis  of  software 
requirements  is  presented  in  Section  3.0. 

4.2.3  Overhead  Processor 

The  computations  and  bookkeeping  required  for  implementing  the  overhead 
functions  can  be  handled  by  the  same  processor  used  for  the  loop  filter.  Memory 
requirements  for  these  features  are  presented  in  Section  3.0.  When  more  than  IK 
bytes  of  program  storage  is  needed,  a  4K  byte  PROM  card  is  added  to  the  system. 

Technology  advances  in  this  area  can  quickly  obsolete  the  results  of 
cost/performance  studies.  As  of  this  writing,  2K  byte  and  4K  byte  PROM  IC's  are 
becoming  available  which  can  replace  the  IK  PROM  on  the  processor  card.  Intel  has 
recently  announced  an  8K  byte  mask  progranmable  read-only  memory  (ROM). 


4.2.4  Local  Reference 

The  assumed  crystal  reference  is  a  5  MHz  oscillator  having  a  drift  of 
1  x  10-10  per  day.  It  Is  a  self-contained,  rack-mountable  unit  with  its  own 
power  supply  and  stand-by  battery  system.  A  voltage  input  of  +5  volts  will 
deviate  the  5  MHz  output  by  +2  x  10-8.  The  short  term  stability  is  1  x 
10-11.  Many  manufacturers,  Including  Hewlett-Packard,  Austron,  Vectron,  and 
Frequency  and  Time  Systems,  Inc.  offer  very  similar  references  of  this  type  with 
approximately  $3,000  price  tags.  A  frequency  doubler  is  used  to  obtain  the  10  MHz. 

The  crystal  reference  is  disciplined  with  the  output  voltage  of  a  D/A 
converter.  A  frequency  resolution  of  at  least  10-H  is  desirable,  since 
quantization  in  a  closed  loop  can  create  limit  cycles  with  amplitudes  of  several 
quantization  steps.  A  resolution  of  10-H  and  a  range  of  4  x  10-8  require 
4,000  quantization  steps  resulting  in  a  12-bit  requirement  for  the  D/A.  This 
range  allows  the  oscillator  to  drift  for  200  days  before  the  center  frequency  must 
be  mechanically  adjusted.  This  is  the  approach  which  was  costed.  Use  of  a  +3  x 
10-7  adjustment  range  with  a  16-bit  D/A  would  permit  the  same  resolution  with  a 
reset  interval  of  8  years,  which  exceeds  the  5  year  MTBF  of  the  reference. 

However,  this  arrangement  would  have  a  sensitivity  of  150  //volt  per  quantization 
step  on  the  voltage  input.  It  would  be  extremely  difficult  to  prevent  noise 
pickup  of  this  amplitude  from  modulating  the  reference. 

A  crystal  oscillator  not  having  a  voltage  control  capability  could  also 
be  disciplined  by  the  phase  microstepper  described  below.  Such  an  approach  would 
undoubtedly  be  more  costly  to  procure,  but  would  possibly  simplify  support 
logistics  by  eliminating  periodic  crystal  readjustments. 

The  assumed  Cesium  Clock  Is  a  Hewlett-Packard  Model  5061A  with  the 
standby  power  supply  and  high  performance  tube  options;  total  price  is  $22,750. 

This  is  a  self-contained  reference  having  an  accuracy  of  +7  x  10-12.  The 
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reference  Is  disciplined  by  shifting  Its  phase  with  an  Austron  2055A  Phase 
Microstepper  costing  $3,550. 

The  Rubidium  standard  Is  a  Hewlett-Packard  Model  5065A  with  the  standby 
power  supply  option;  total  cost  Is  $8,575.  This  is  a  self-contained  reference 
having  a  drift  of  +1  x  10-H  per  month.  For  disciplined  approaches,  its  phase 
is  shifted  with  the  Phase  Microstepper. 

4.2.5  Overhead  Interface 

The  Transmitter  and  Receiver  portions  of  the  Overhead  Interface  are 
shown  in  Figure  4.2.5.  A  total  of  55  bits  of  information  Is  required  for 
implementing  all  optional  features.  Triplicating  this  figure  results  in  165. 
Adding  35  bits  for  framing  and  time  of  day  information  results  in  a  total  of  200 
bits.  Transmitting  this  information  1.5  times  per  second  results  in  a  300  b/s 
data  stream  which  may  be  transmitted  via  a  standard  300  b/s  channel  of  the  LSTDM 
and  the  Digital  Radio  Orderwire.  This  information  would  only  occupy  0.16  percent 
of  the  192K  bandwidth  allocated  for  the  orderwire. 

The  Transmitter  accepts  the  information  words  8  bits  at  a  time  from  the 
8080,  serially  transmits  each  word  three  times,  and  generates  periodic  framing 
information  to  allow  separation  of  data  on  the  receiving  end.  The  Transmitter 
occupies  one  PC  card. 

The  Receiver  converts  the  serial  stream  to  parallel,  accumulates  three 
successive  words,  votes  to  correct  errors,  and  presents  the  parallel  data  to  the 
8080.  A  sync  correlator  detects  the  presence  of  the  sync  pattern  and  initializes 
the  sequencer  to  pick  out  the  words  at  the  proper  time.  The  Receiver  occupies  two 
PC  cards. 

4.2.6  Frequency  Synthesizers  and  Distribution  Amplifiers 

The  function  of  the  frequency  distribution  system  Is  to  generate 
phase-related  rates  to  clock  all  equipments  which  are  to  operate  synchronously 
with  the  local  clock.  Such  equipments  may  include  all  devices  in  the  transmit 
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Figure  4.2.5.  Overhead  Interface 
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hierarchy  from  data  terminal  equipment  to  the  digital  Radio,  and  all  devices  lower 
in  the  receive  hierarchy  than  the  elastic  buffers.  i 

DCEC  specification  R220-77-2  describes  such  a  system  which  operates 
from  the  1  MHz  outputs  of  the  AN/GSQ-183  Loran  Receiver.  Table  4.2.6  is  a  list  of 
rates  from  that  specification  along  with  the  number  of  required  outputs  of  each 
rate.  A  balanced  low-level  driver  (MIL-188-114)  must  be  used  for  each  output. 

This  system  is  being  procured  for  use  in  the  interim  communication  network  (DCS 
II),  and  might  possibly  be  useable  for  the  future  DCS. 

The  assumed  design  generates  the  rates  of  Table  4.2.6  from  the  output 
of  the  10  MHz  local  reference.  Each  of  these  rates,  as  well  as  the  10  MHz,  is  a 
multiple  of  8,000  b/s.  The  phase  of  these  rates  should  be  such  that  if  each  one 
(including  the  10  MHz  input)  is  divided  down  to  8,000  b/s,  then  the  8,000  b/s 
waveform  from  each  countdown  chain  should  be  in  phase.  The  rising  edges  of  this 
8,000  b/s  waveform  (or  a  submultiple  thereof)  should  be  used  to  initiate  the  frame 
departure  in  the  TD-1193  and  Digital  Radio.  The  importance  of  this  concept  cannot 
be  overstressed.  The  8,000  b/s  waveform  (or  a  submultiple)  should  be  referred  to 
as  the  local  clock  rather  than  the  10  MHz.  The  10  MHz  provides  a  means  of 
interpolating  between  pulses  of  this  local  clock  to  provide  a  time  resolution  of 
100  nsec  (but  possibly  even  a  better  accuracy). 

Realistically,  the  individual  8,000  b/s  waveforms  counted  down  from 
each  rate  will  be  slightly  out  of  phase  with  each  other.  Since  the  time  of 
departure  of  the  12.928  Mb/s  waveform  of  the  TD-1193  is  the  event  by  which 
adjoining  nodes  will  measure  the  local  node's  time,  the  8,000  b/s  counted  down 
from  the  waveform  should  agree  as  closely  with  the  local  clock  as  possible. 

Equipment  error  contributions  of  less  than  approximately  +60 
nanoseconds  per  link  have  been  suggestedl75  a$  desirable  for  maintaining  a 
precise  time  error  of  1  microsecond  throughout  the  network.  This  allows  a  30  nsec 
tolerance  on  each  end  of  the  link.  Some  error  contributions  will  be  generated  by 
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the  radios,  multiplexers,  and  phase  detectors,  as  well  as  in  the  generation  of  the 
12.928  Mb/s  described  above.  A  10  nanosecond  phase  error  between  the  individual 
8,000  b/s  waveforms  might  be  a  desirable  goal.  It  is  not  known  if  these 
considerations  were  made  for  the  system  presently  being  procured. 

Figure  4.2.6  depicts  the  assumed  design  of  the  frequency  distribution 
system.  A  minimum  of  three  VCXO's  must  be  used:  2048  kHz  (which  can  furnish 
submultiples  down  to  16  kHz),  1544  kHz,  and  38.784  MHz  (which  can  be  divided  down 
into  each  of  the  four  TD-1193  rates).  The  usual  procedure  would  be  to  divide  each 
VCXO  output  down  to  8  kHz  and  measure  the  phase  difference  of  these  waveforms  with 
the  8  kHz  divided  down  from  the  10  MHz  reference.  With  this  procedure,  a  1  part 
per  1000  phase  angle  error  (which  might  be  difficult  to  achieve)  would  result  in 
an  unacceptable  +625  nsec  absolute  error  which  could  wander  around  with 
temperature. 

To  improve  the  problem,  an  intermediate  512K  is  created  by  a  64  MHz 
VCXO  locked  to  the  10  MHz  (phase  comparisons  performed  at  2  MHz).  The  remaining 
three  VCXO's  are  then  locked  with  phase  comparisons  made  at  this  intermediate 
rate.  The  T1  rate  and  the  TD-1193  rates  must  then  originate  from  98.816  MHz  and 
155.136  MHz  VCXO's  respectively.  This  will  result  in  no  more  than  +10  nsec  phase 
errors  even  with  1  percent  phase  angle  measurement  accuracy.  Some  ECL  was  used  in 
this  design.  No  redundant  elements  were  used  in  this  subsystem.  Triple  redundant 
synthesizer  modules  (as  required  in  the  aforementioned  specification)  will 
increase  costs  proportionately.  Each  module  requires  1  PC  card. 

The  line  drivers  were  implemented  with  commercial  devices  having 
voltage  swings  similar  to  those  of  MIL-188-114  balanced  drivers.  Devices  which 
conform  rigidly  to  the  MIL-188-114  specification  are  not  commercially  available 
and  must  be  special-made.  Some  companies  (such  as  Sperry)  have  developed  hybrid 
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Table  4.2.6.  Clock  Frequencies  and  Maximum  Number  of 
Outputs  Per  Frequency 


Maximum  Number  of 

Clock  Frequency  Rates  Outputs  per  Frequency  Rate 


16  kHz 

25 

32  kHz 

10 

56  kHz 

15 

64  kHz 

10 

128  kHz 

15 

256  kHz 

15 

5  »  kHz 

10 

1024  kHz 

10 

1544  kHz 

60 

2048  kHz 

10 

3.232  MHz 

10 

6.464  MHz 

10 

9.696  MHz 

10 

12.928  MHz 

10 

circuits  which  can  be  produced  on  special  order. 
4.2.6  implies  that  220  such  devices  are  required. 
18  PC  cards  in  the  assumed  design. 


Typical  cost  is  $80  each.  Table 
This  number  of  drivers  required 
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4.2.7  Elastic  Buffer 

The  Elastic  Buffer  absorbs  rate  variations  between  received  data  and 
the  local  clock.  They  may  be  placed  anywhere  in  the  Demultiplexer  hierarchy  as 
long  as  the  point  where  timing  is  derived  is  higher  in  the  hierarchy  than  the 
buffers.  Demultiplexers  higher  in  the  hierarchy  than  the  buffers  derive  their 
timing  from  the  associated  incoming  links,  and  those  lower  in  the  hierarchy 
receive  timing  from  the  nodal  clock. 

Since  all  demultiplexers  lower  in  the  hierarchy  than  the  buffers  are 
synchronous  with  each  other  and  with  the  multiplexers,  channel  outputs  from  such 
demultiplexers  may  be  routed  to  channel  inputs  on  any  multiplexer  for 
retransmission  on  another  link.  This  is  a  strong  argument  for  placing  the  buffers 
as  high  in  the  hierarchy  as  possible.  Placing  them  between  the  radio  and  the 
TD-1193  makes  tandeming  at  any  level  possible,  including  routing  one  of  the  radio 
channels  directly  to  another  radio  for  retransmission.  There  is  another  advantage 
to  this  approach  which  many  people  do  not  realize.  A  multiplexer/ demultiplexer 
set  can  be  built  less  expensively  if  the  MUX  and  DEMUX  do  not  have  to  operate  from 
independent  timing. 

The  12.928  MHz  data  rate  from  the  Digital  Radio  poses  no  difficult 
design  problem  for  the  Elastic  Buffer.  For  an  independent  clock  approach 
employing  Cesium  clocks  and  a  24-hour  buffer  reset  interval,  the  required  buffer 
size  (at  12.928  MHz)  would  be  46  bits.  For  a  Rubidium  clock  with  a  10"^  per 
month  drift,  6  month  recalibration  interval,  and  24-hour  buffer  reset  interval, 
the  required  buffer  size  is  270  bits.  If  disciplined  nodes  containing  crystal 
clocks  can  be  controlled  to  within  10  jis  of  the  Master  (within  20  ps  of  each 
other)  then  +259  =  518  bits  are  required  during  normal  operation.  A  node  with  a 
good  crystal  clock  will  drift  an  additional  +56  bits  during  the  first  24  hours 
after  being  severed  from  the  network.  Provided  this  is  enough  time  to  get  the 
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link  back  up,  a  total  of  630  bits  are  required  based  on  the  above  assumptions. 

The  results  of  the  simulations  were  not  available  at  the  time  of  this  analysis. 
Recent  results  indicate  that  for  some  approaches,  larger  phase  differences  than 
those  assumed  here  may  occur.  As  of  this  writing  no  attempt  has  been  made  to  cost 
a  larger  buffer. 

Figure  4.2.7  is  a  block  diagram  of  a  1024-bit  buffer  designed  to 
operate  at  12.928  MHz.  For  relatively  small  buffer*,  commercially  available  FIFO's 
(First-In-First-Out  Memories)  are  perhaps  the  best  approach.  These  IC's  contain 
the  control  circuitry  for  moving  data  bits  forward  whenever  one  is  extracted  from 
the  output.  The  FIFO  in  this  configuration  need  only  operate  at  1.616  MHz.  Bits 
from  the  12.928  MHz  stream  are  serially  accumulated  and  stored  broadsize  (eight  at 
a  time)  at  1/8  the  original  rate.  The  FIFO  array  can  be  implemented  with  either 
four  64  x  4  IC's  or  with  four  32  x  8  IC's.  Both  types  are  available  which  can 
operate  at  the  indicated  rates.  Handshaking  signals  are  available  to  allow  the 
IC's  to  be  cascaded.  The  additional  circuitry  in  Figure  4.2.7  is  required  to 
initialize  the  buffer  (inhibit  output  clock  until  it  half  fills)  and  to  monitor 
overflow  and  underflow.  Two  of  these  buffers  occupy  one  PC  card. 

4.3  LIFE  CYCLE  COST  ANALYSIS 

4.3.1  Configurations  for  Costing 

It  was  desired  to  determine  the  cost  of  adding  each  one  of  the  optional 
features  to  a  basic  timing  approach.  However,  it  is  not  possible  to  examine  the 
cost  of  each  feature  individually.  Rather,  due  to  commonality  of  required 
components,  it  is  more  feasible  to  cost  all  practical  configurations  which  Include 
the  features  In  various  combinations.  The  list  of  configurations  simulated  in 
Section  3.0  was  chosen  for  costing,  so  that  performance  versus  cost  trade-offs  may 
be  made. 
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It  became  readily  apparent  that  many  of  the  configurations  from  Section 
3.0  (see  Table  3.2.3.11)  were  practically  identical  with  respect  to  cost. 
Estimating  cost  differences  between  such  configurations  was  beyond  the  precision 
of  the  methods  used  for  costing,  and  such  configurations  were  combined  into  one. 
This  resulted  in  reducing  the  16  configurations  of  Table  3.2.3.11  into  ten 
slightly  more  general  configurations.  The  independent  clock  approach  was  added 
for  completeness,  resulting  in  eleven  configurations  which  were  costed.  These 
configurations  are  listed  in  Table  4.3.1  using  the  same  configuration  numbers  as 
in  Table  3.2.3.11  for  clarity. 

Table  4.3.1.  Configurations  for  Costing 

1-2.  Directed  Control  (Single-Ended) 

3-7.  Mutual  Control  (Single-Ended) 

8.  Directed  Control,  Double-Ended 

9-10.  Mutual  Control,  Double-Ended 

11.  Directed  Control,  Double-Ended,  Independence  of  Measurement  and 
Correction 

12.  Directed  Control,  Double-Ended,  Independence  of  Measurement  and 
Correction,  Phase  Reference  Combining 

13.  Directed  Control  (Single-Ended),  Self-Organizing 

14.  Directed  Control,  Double-Ended,  Self-Organizing 

15.  Add  Self-Organizing  to  Configuration  11 

16.  Add  Self-Organizing  to  Configuration  12 

17.  Independent  Clocks 
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4. 3.1.1  Software  Configurations 

Paragraph  3.3.11  presented  an  analysis  of  software  requirements  in 
terms  of  bytes  of  storage  for  the  8080  microprocessor.  Software  costs  were 
computed  based  on  number  of  lines  of  code.  A  line  of  8080  code  expands  into  one, 
two,  or  three  bytes  (or  no  bytes  if  a  comment  statement)  with  the  average  being 
slightly  greater  than  2  bytes  per  line. 

Twenty  percent  was  added  to  the  requirements  presented  in  Paragraph 
3.3.11  to  accommodate  diagnostic  software.  Table  4. 3. 1.1  lists  the  software 
requirements  in  terms  of  bytes  and  lines  of  code.  For  configurations  requiring 
more  than  IK  bytes  of  memory,  the  PROM  card  is  added  to  the  system. 


Table  4. 3. 1.1.  Software  Requirements 


Configuration  Lines  Bytes 


1-2 

Dir. 

Con., 

Sngl . 

240 

500 

3-7 

Mut. 

Con., 

Sngl. 

360 

760 

8 

Dir. 

Con. , 

Dble. 

440 

920 

9-10 

Mut. 

Con., 

Dble. 

630 

1320 

11 

Dir. 

Con., 

Dble., 

IMC 

480 

1010 

12 

Dir. 

Con., 

Dble., 

IMC, 

PRC 

1100 

2310 

13 

Dir. 

Con. , 

Sngl., 

SO 

420 

880 

14 

Dir. 

Con., 

Dble. , 

SO 

630 

1320 

15 

Dir. 

Con. , 

Dble., 

IMC, 

SO 

670 

1410 

16 

Dir. 

Con., 

Dble., 

IMC, 

PRC,  SO 

1300 

2730 

17 

Ind. 

Clk. 

0 

0 

Legend _ 

Dir.  Con.  -  Directed  Control 
Mut.  Con.  -  Mutual  Control 
Ind.  Clk.  -  Independent  Clock 
Dble.  -  Double-Ended 

IMC  -  Independence  of  Measurement  and  Correction 

PRC  -  Phase  Reference  Combining 

SO  -  Self-Organizing 
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4. 3.1. 2  Hardware  Configurations 

As  previously  stated,  the  synthesizers,  distribution  amplifiers,  and 
buffers  were  placed  in  a  separate  drawer  (the  Basic  Drawer).  This  drawer  is 
Included  in  all  configurations.  Table  4. 3. 1.2-1  is  a  breakdown  of  its  contents. 


Table  4. 3. 1.2-1.  Basic  Drawer  Contents 


Component 

Ave./Box 

Max/Box 

Synthesizer  1 

1 

1 

Synthesizer  2 

1 

1 

Synthesizer  3 

1 

1 

Synthesizer  4 

1 

1 

Distribution  Amp. 

18 

18 

Dual  Buffer 

4 

7 

Motherboard 

1.5 

1.5 

Supply  (+5) 

1 

1 

Supply  (-5) 

1 

1 

Supply  (+12) 

1 

1 

Drawer 

1 

1 

A  fractional  motherboard  indicates  only  part  of  it  is  wired.  The  motherboard 
wiring  and  power  supplies  will  support  the  maximum  (Max/Box)  number  of  PC  cards 
expected.  Costs  were  based  on  the  average  number  of  PC  cards. 

All  configurations,  except  Independent  Clocks,  contain  a  second  drawer 
(the  Disciplining  Drawer).  Table  4. 3. 1.2-2  is  a  breakdown  of  the  components 
contained  in  this  drawer  for  all  configurations.  The  motherboards  and  power 
supplies  vary  in  size  with  the  configurations,  and  in  each  case  support  the 
maximum  number  of  links.  Table  4. 3. 1.2-3  is  a  breakdown  of  the  PC  cards  whose 
quantities  vary  between  configurations.  Costs  are  based  on  the  average  number. 
The  choices  of  numbers  in  some  cases  are  rather  subjective  and  are  based  on 
assumptions  of  the  number  of  neighbors  higher,  equal,  or  lower  in  the  hierarchy. 
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Table  4. 3. 1.2-2.  Disciplining  Drawer  Components 


Component 


Quantity 


Processor 

Controller 

D/A 

Supply  (+5) 
Supply  (+121 
Supply  (+15) 
Motherboard 
Drawer 


1 

1 

1 

1  (Size  Varies) 
1 
1 

Varies 

1 


Table  4. 3. 1.2-3.  Disciplining  Drawer  Optional  Components 


Configuration 


1-2 

3-7 

8 

9-10 

11 

12 

13 

14 

15 

16 
17 


Sync.  Ex. 
Ave.  Max. 


Ph.  Pet. 
Ave.  Max. 


OH  Rev. 
Ave.  Max. 


1  1 

4  7 

4  7 

4  7 

4  7 

4  7 

1  1 

4  7 

4  7 

4  7 


1  1  0 

4  7  0 

4  7  1 

4  7  4 

4  7  1 

4  7  4 

1  1  4 

4  7  4 

4  7  4 

4  7  4 


Not  Applicable 


0 

0 

1 

7 

1 

7 

7 

7 

7 

7 


Legend _ 

Sync.  Ex.  -  Sync  Extractor  (1  card) 

Ph.  Det.  -  Phase  Detector  (2  cards) 

OH  Rev.  -  Overhead  Receiver  (2  cards) 

OH  Xmit  -  Overhead  Transmitter  (1  card) 

PROM  -  Additional  4K  bytes  memory  (1  card) 


OH  Xmit 


Ave.  Max. 

0  0 

0  0 

3  6 

4  7 

3  6 

4  7 

4  7 

4  7 

4  7 

4  7 


PROM 


0 

0 

0 

1 

0 

1 

0 

1 

1 

1 
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4.3.2  Costing  Methodology 

The  Software  and  Hardware  costing  were  performed  separately.  The  RCA 
PRICE  program  was  used  for  Hardware  costs.  At  the  time  of  the  costing,  the  PRICE 
Software  model  was  not  available.  The  Life-Cycle  Cost  of  a  system  is  divided  into 
three  parts:  Development,  Production  and  Maintenance  for  the  life  of  the 
equipment. 

Development  costs  include  equipment  design  and  construction  of 
prototypes.  These  costs  are  nonrecurring;  i.e.,  they  are  independent  of  the 
quantity  of  systems  to  be  built.  Production  costs  include  tooling  up  for 
production,  material  and  labor  for  building  each  system,  and  labor  for  testing 
finished  systems.  Tooling  includes  procuring  or  building  special  equipment  used 
for  fabricating  and  testing  the  systems.  Production  costs  are  proportional  to  the 
quantity  of  systems,  but  the  relation  is  not  linear.  Due  to  a  "learning  curve," 
the  cost  of  production  on  per  system  basis  decreases  with  the  number  of  systems. 
Much  of  the  tooling  is  up  front;  however,  retooling  generally  is  necessary  due  to 
wear  and  breakage.  Maintenance  costs  include  test  and  repair  labor  costs, 
transportation,  supply  management,  and  purchase  of  piece  parts. 

Acquisition  costs  simply  consist  of  development  costs  plus  the  costs  to 
produce  the  desired  number  of  systems.  When  considering  Life-Cycle  Costs,  the 
Production  Costs  are  modified  to  include  production  of  spares  and  production  (or 
purchase)  of  test  equipment  to  support  the  system  in  the  field  for  a  specified 
number  of  years. 

4.3.2. 1  Software  Model 

Generally,  software  costs  are  considered  to  be  nonrecurring.  The  costs 
simply  consist  of  writing  the  code.  If  the  software  is  the  same  in  all  systems, 
the  software  costs  are  independent  of  the  number  of  systems.  A  reasonable  cost 
for  developing  code  of  this  type  is  $15  per  line.  For  the  system  under 
consideration,  the  programs  must  be  "burned"  into  the  PROM  IC's  for  each 
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8080  or  PROM  card.  This  process  is  mechanized  and  may  be  considered  to  be  part  of 
the  fabrication  process.  These  costs  are  included  in  the  hardware  production 
costs  by  assuming  the  complexities  of  these  cards  to  be  slightly  higher  than 
otherwise  would  have  been  assumed.  Thus  no  additional  production  costs  for 
software  were  assumed. 

Software  maintenance  is  a  euphemism  invented  in  recent  years  to 
describe  the  costs  of  continually  rewriting  programs  which  were  not  written 
properly  in  the  first  place.  Many  of  these  "errors"  result  from  simply  not 
anticipating  every  possible  situation  with  which  the  software  might  have  to  deal. 
Even  after  "thorough"  testing,  residual  errors  may  become  apparent  only  after  very 
long  periods  of  operation.  As  a  result,  some  software  support  may  be  required  for 
the  life  of  the  system.  A  figure  of  $5  per  line  per  year  has  been  determined  as  a 
typical  figure  for  such  support. 

4. 3. 2. 2  Hardware  Model 

The  Life-Cycle  Costs  for  the  hardware  configurations  were  computed 
using  the  RCA  PRICE  (Programmed  Review  of  Information  for  Costing  and 
Evaluation).  There  are  two  programs  involved:  PRICE  83B  computes  Acquisition 
Costs,  and  Price  LI  modifies  the  production  costs  and  adds  in  maintenance  to 
complete  the  Life-Cycle  Costs.  PRICE  is  a  proven  parametric  cost  estimating  model 
which  provides  reliable  estimates  of  system  acquistion  costs  (development  and 
production).  The  PRICE  838  program  generates  design  to  unit  production  cost, 
based  upon  variations  in  designs,  performance,  schedules,  reliability,  economic 
escalations,  etc.  The  price  inputs  are  primarily  physical  characteristics  of  the 
design  concept.  These  include  weight,  volume,  manufacturing  complexity,  platform, 
quantity,  development  schedule  and  production  schedule.  The  outputs  feature 
recurring  and  nonrecurring  costs  for  development  and  production  as  well  as  a  unit 
production  cost  value  for  each  entry.  PRICE  83B  also  develops  inputs  for  the 
price  Ll  model  for  Life-Cycle  Cost  (LCC). 
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Figure  4. 3. 2. 2  shows  the  operation  and  interrelationship  of  the  PRICE 
cost  model  (PRICE  83B)  and  the  PRICE  Life-Cycle  Cost  Model  (PRICE  Ll).  The 
following  paragraphs  give  an  overview  of  the  approach  taken  when  using  the  RCA 
model.  Physical  characteristics  are  used  to  describe  the  item  to  be  inputted  into 
the  PRICE  83B  cost  model.  The  following  Inputs  were  required  to  effectively 
utilize  the  price  program: 

1.  Qty  -  Total  amount  of  production  units. 

2.  Protos  -  Total  amount  of  non  production  units,  i.e.,  EDM  STM, 
Mockups,  etc.  Any  equivalent  units,  including  partial  units,  are 
i nc 1 uded . 

3.  Wt  -  Total  weight  in  pounds  of  the  assembly  to  be  priced.  The 
weight  is  for  one  each  assembly.  Weight  includes  enclosure  and 
electronics. 

4.  Vol  -  Volume  in  ft3  of  the  subject  unit.  For  electro/mechanical 
assemblies,  the  volume  is  a  critical  parameter  and  must  be  accurate. 

5.  Qtysys  -  How  many  of  the  subject  items  are  required  on  a  per  system 
basis? 

6.  Integration  -  A  description  of  the  structural  and  electrical 
integration  of  the  subject  unit  to  the  next  higher  assembly. 

Examples  of  types  of  integration: 

•  Power  furnished 

•  Power  furnished  to  assembly  plus  cabled  output 

•  Power  furnished  to  assembly  plus  calibration  and  adjustment 
(tuning) 

•  Power  furnished  to  assembly,  cabled  output,  calibration  and/or 
tuning  required  plus  possible  parts  assembly  replacements 
including  wiring  and  corrections 

•  Physical  mounting  on  one  side  only 


3706 A 


4-32 


t  Physical  mounting  on  more  than  one  side,  no  machining  required 

•  Physical  mounting  requiring  considerable  interface  machining 

7.  Structure  weight  -  Total  weight  of  the  enclosure,  including  the 
following  items: 

«  Metal  sides 

•  Screws,  nuts,  bolts,  etc. 

•  Connectors 

•  Magnetics 

•  Partitions 

•  Heat  sinks 

•  Stiffening  members 

•  Gears,  relays,  and  electromagnetic  devices 

8.  Structure  Description  -  The  description  includes,  but  is  not 
limited  to,  shape,  type  of  material,  environment,  construction 
technique,  i.e.,  costing,  machining,  etc. 

9.  Newness  of  Structural  Design  -  The  answer  to  this  item  should  be 
indicated  as  a  percent  with  100  percent  being  a  totally  new  design 
with  no  existing  documentation  or  supporting  analysis. 

10.  Useful  Volume  for  Electronics  -  Defined  as  a  percentage  of  volume 
the  space  available  for  electronic  packaging. 

11.  Electronic  Description  -  Describe  the  electronic  function  of  the 
subject  item.  Included  in  this  description  are  card  types, 
quantity  and  sizes.  The  circuitry  and  component  type  is  also 
required.  In  addition  to  card  quantity,  the  number  of  unique  cards 
or  functions  are  described. 

12.  Newness  of  Electrical  Design  -  The  percentage  of  new  design 
required.  This  is  a  low  number  if  some  of  the  design  has  been  done 
on  another  program.  It  is  a  high  number  for  totally  new  design. 
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13.  Expected  power  dissipation  In  Watts  as  it  applies  to  the  subject 
unit. 

14.  Component  Quantity  -  The  estimated  total  number  of  electronic 
components  excluding  relays  and  other  electro/magnetic  devices. 

15.  Development  and  Production  Schedule  Data 
Engineering  Development  Schedule 

•  Calendar  year  In  which  program  starts  (Not  FY  Year). 

•  Name  of  month  during  first  year  in  which  design  activity  begins. 

•  Number  of  months  ARO  to  completion  of  the  first  qualification 
tested  unit. 

•  Number  of  months  ARO  to  completion  of  testing  of  the  last 
development  or  prototype  unit. 

Production  Schedule 

•  Total  number  of  months  from  the  first  month  of  the  calendar 
year  that  significant  production  activity  Is  initiated. 

•  Total  number  of  months  from  the  first  month  of  the  calendar 
that  the  last  production  unit  will  be  delivered. 

16.  Engineering  Interpretation  of  design  complexity  -  One  of  the  PRICE 
parameters  requires  the  users  to  catalog  the  design  activity  of  the 
subject  unit  to  be  either,  2nd  Generation,  New  or  advance  in  State 
of  the  Art. 

17.  Engineering  Change  Activity  During  Production  -  Values  are  inputted 
as  a  percent  of  documentation  regeneration  due  to  schedule 
legislation,  testing  or  experience  with  customer  redirection. 

18.  Tooling  requirements  are  generated  as  routine,  significant  or 
nonexistent. 
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19.  Production  Methods  -  A  description  of  the  methodology  to  be 


employed  in  the  assembly  and  fabrication  of  the  subject  unit  is 
required.  This  data  will  determine  the  type  of  learning  curve 
during  production. 

20.  Systems  Management  and  Data  Management  -  A  brief  description  of  the 
System  Management  role  and  characterization  of  the  CDRL  items  are 
required. 

21.  Platform  is  the  variable  that  controls  the  environmental, 
reliability,  impact  and  application  impact  on  the  design. 

4. 3. 2. 3  Program  Assumptions 

The  Life-Cycle  Costs  of  a  system  of  200  nodes  were  computed  based  on  a 
system  lifetime  of  20  years.  The  software  costs  were  computed  by  the  simple 
formulas  stated  in  Paragraph  4. 3. 2.1.  Table  4. 3. 2. 3-1  is  a  breakdown  of  the 
software  costs  for  the  various  configurations. 

Hardware  costs  were  computed  by  a  very  detailed  process.  The  hardware 
was  described  in  detail  to  PRICE  83B  which  computed  Acquistion  costs.  This 
process  was  performed  by  personnel  who  are  very  experienced  with  the  operation  of 
the  PRICE  Model.  These  descriptions  were  performed  at  the  PC  card  level  and  were 
checked  for  reasonability.  The  PC  cards  were  combined  in  the  various  combinations 
with  purchase  items  (items  such  as  power  supplies  and  references  for  which  catalog 
prices  were  used)  to  form  the  different  configurations.  Another  output  of  PRICE 
83B  is  the  LC  file  which  includes  unit  costs,  computed  MTBF  and  MTTR  values,  and 
other  pertinent  factors  used  by  PRICE  LI  to  compute  the  Life-Cycle  Costs. 

Editing  the  LC  file  gives  the  user  the  opportunity  to  provide  all  the 
information  he  can  about  the  system.  For  example,  the  predicted  MTBF's  of 
purchase  items  were  overridden  at  this  point  with  actual  values  from  the 
manufacturer.  MTBF's  for  designed  equipment  were  also  checked  and  altered  if 
unreasonable. 


— 
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Table  4. 3. 2. 3-1  Software  Costs  (Thousands) 


Configuration 

Development 

Maintenance 

1-2  Dir.  Con.  (Sngl.) 

3.6 

24 

3-7  Mut.  Con.  (Sngl.) 

5.4 

36 

8  Dir.  Con. ,  Dble. 

6.6 

44 

)-10  Mut.  Con.,  Dble. 

9.5 

63 

11  Dir.  Con.,  Dble., 

ICEM&C 

7.2 

48 

12  Dir.  Con.,  Dble., 

ICEM&C, 

PRC 

16.5 

110 

13  Dir.  Con.  (Sngl.) 

,  SO 

6.3 

42 

14  Dir.  Con.,  Dble., 

SO 

9.5 

63 

15  Dir.  Con.,  Dble., 

ICEM&C, 

SO 

10.1 

67 

16  Dir.  Con.,  Dble., 

ICEM&C, 

PRC,  SO 

19.5 

130 

17  Ind.  Clk. 

0 

0 

The  LI  Model  was  then  exercised  on  the  LC  file  using  the  force  structure  of 
Fig.  4. 3. 2. 3.  The  support  philosophy  adheres  to  DOD  Directive  4151-16  which 
states  that  there  shall  be  three  echelons  of  support.  Simple  repair  is  performed 
at  the  organizational  shop,  and  consists  of  fault  isolation  to  an  LRU  (Line 
Replaceable  Unit),  replacement  with  a  spare  and  shipment  to  a  higher  level  for 
repair.  Generally,  repair  to  piece  part  is  performed  at  an  Intermediate  shop  if 
not  too  complex,  and  at  a  Depot  otherwise.  Since  most  DCS  nodes  will  probably  be 
located  at  major  military  installations,  the  Intermediate  shop  is  considered  to  be 
a  general  repair  shop  local  to  the  base.  It  was  assumed  that  25X  of  the  nodes 
would  be  remote,  so  that  a  total  of  150  Intermediate  shops  were  used.  A  Depot  was 
assumed  for  each  of  the  three  services  who  will  support  DCS. 
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ORGANIZATIONAL 

(200) 


INTERMEDIATE 

(150) 


DEPOT  (3) 


•  SUPPORT  YEARS  -  20 

•  EQUIPMENT  USAGE  -  730.5  HOURS/MONTH  (CONTINUOUSLY) 

•  PRICE  83B  CHOOSES  MTBF  FOR  DESIGNED  ITEMS 

•  CATALOG  VALUES  USED  FOR  MTBF  FOR  PURCHASED  ITEMS 

•  FOR  RELIABLE  ITEMS,  NO  SPARES  AT  INTERMEDIATE 

10190-20 


Figure  4. 3. 2. 3.  LCC  Assumptions 
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Table  4. 3. 2. 3-2  Is  a  breakdown  of  the  components  costs  for 
Configuration  16  (TRD  using  crystal  clocks).  Table  4. 3. 2. 3-3  gives  the  Life-Cycle 
Costs  of  the  various  configurations.  These  costs  are  based  on  crystal 
clocks  for  the  disciplined  approaches.  Configuration  16  (TRD)  is  repeated  for 
Cesium  and  Rubidium.  Configuration  17  is  also  given  for  both  Cesium  and  Rubidium 
but  not  Crystal. 

4.4  Conclusions 

The  prices  obtained  in  Table  4. 3. 2. 3-3  may  be  compared  with  the 
simulation  results  for  cost  versus  performance  trade-offs.  Comparing  the  costs  of 
disciplined  approaches  using  crystal  clocks,  they  are  all  very  close  (about  25 
percent  total  variation).  A  surprising  result  was  that  most  disciplined 
approaches  using  crystal  clocks  came  out  slightly  higher  than  Independent  clocks 
using  Cesium  clocks.  This  is  surprising  in  light  of  the  fact  that  the  Independent 
Cesium  clock  approach  has  a  higher  acquisition  cost.  A  probable  reason  for  this 
result  is  that  commercial  parts  were  assumed  for  all  designed  equipment,  and  the 
costs  for  maintaining  the  Discipling  Drawer  offset  the  high  cost  of  the  Cesium 
clock. 
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Table  4.3. 

2.3-2. 

Hardware  Cost  Breakdown 

i  for  Configuration  16 

Qty 

Dev 

Prod 

Support 

Total 

Synth  1 

1 

29 

72 

185 

287 

Synth  2 

1 

31 

77 

195 

303 

Synth  3 

1 

46 

71 

188 

305 

Synth  4 

1 

31 

75 

193 

298 

Dist  Amp 

18 

14 

399 

1403 

1816 

El  as  Buf 

4 

21 

162 

519 

702 

Mother 

1 

65 

458 

2 

525 

Power  3 

1 

0 

116 

138 

255 

Power  4 

1 

0 

137 

144 

281 

Power  8 

1 

0 

24 

75 

99 

Drawer 

1 

2 

647 

163 

811 

I&T* 

1 

6 

430 

96 

532 

Subtotals 

245 

2668 

3301 

4214  (Basic  Box) 

PCI 

4 

18 

103 

305 

426 

PC2 

4 

19 

103 

329 

450 

Syncex 

4 

33 

117 

406 

556 

0HRCV1 

4 

28 

141 

454 

623 

0HRCV2 

4 

20 

93 

288 

401 

OHXMTR 

4 

23 

105 

360 

488 

PROM 

1 

14 

68 

132 

214 

8080 

1 

11 

60 

145 

216 

D/A 

1 

48 

90 

224 

361 

Timer 

1 

24 

59 

142 

225 

Power  5 

1 

0 

196 

154 

351 

Table  4. 3. 2. 3-2.  Hardware  Cost  Breakdown  for  Configuration  16  (Continued) 


flt* 

Dev 

Prod 

Support 

Total 

Power  9 

1 

0 

85 

131 

216 

Power  10 

1 

0 

37 

115 

152 

Mother 

1 

49 

375 

2 

427 

Drawer 

1 

0 

647 

163 

810 

I&T* 

1 

6 

157 

324 

487 

Subtotals 

293 

2436 

3674 

6403  (Disc  Box) 

XTAL 

1 

3 

1281 

129 

1414 

I&T* 

1 

12 

416 

142 

570 

Totals 

553 

6801 

7246 

14601 

♦Note:  I&T  means  Integration 

and  Test 
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BENEFIT/PENALTY  OF  PRECISE  TIME  AVAILABILITY 


This  section  considers  the  potential  benefits  of  having  precise  time 
(Coordinated  Universal  Time  ( UTC ) )  traceable  to  the  United  States  Naval 
Observatory,  (USNO)  available  to  each  of  the  major  nodes  for  the  synchronization 
of  the  Defense  Communications  System  (DCS),  for  intranetwork  synchronization  and 
for  internetwork  operability.  Some  of  the  benefits  to  be  discussed  will  be 
directly  related  to  the  fact  that  the  time  is  precise  at  various  modes  in  the 
network.  In  a  number  of  other  cases  the  benefits  result  more  directly  from  the 
MEANS  BY  WHICH  precise  time  is  disseminated  and  only  indirectly  because  it  is 
precise  time  that  is  the  source.  Internetwork  synchronization  and  operability 
deals  with  interfacing  the  DCS  network  timing  subsystem  to  commercial 
communications  systems  and  other  Government  timing  subsystems  presently 
implemented  or  being  implemented.  Examples  of  other  timing  subsystems  are 
navigation  systems  such  as  Loran-C,  Global  Positioning  System,  Omega  and  Transit, 
and  examples  of  other  communications  systems  include  TRI-TAC,  NATO,  DSCS  and  AT&T. 

Paragraph  5.1,  Preliminary  Concepts,  is  devoted  to  a  discussion  of 
Coordinated  Universal  Time  (UTC)  and  the  relationship  of  the  timing  subsystem 
features  described  in  Paragraph  2.2  to  precise  time. 

Paragraph  5.2  discusses  the  benefits  of  the  availability  of  precise 
time  for  DCS  network  synchronization.  This  paragraph  considers  the  problem  as  a 
communications  network  synchronization  problem  and  the  systems  benefits  to  commu¬ 
nications  network  synchronization  are  stressed. 

In  Paragraph  5.3  potential  benefits  of  the  availability  of  precise  time 
to  DOD  intrasystem  operability  are  evaluated.  Mainly,  the  synchronization  of 
equipment  is  considered. 

The  potential  benefits  of  the  availability  of  precise  time  to  DOD 
intersystems  applications  are  evaluated  in  Paragraph  5.4.  The  prime  benefit  here 
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Is  Increased  survivability  through  interoperability.  Networks  which  disseminate 
precise  time  traceable  to  the  USNO  have  more  potential  to  interoperate  with  the 
DCS.  This  improves  the  survivability  of  both  networks  during  periods  of  military 
stress.  Commercial  networks  that  have  the  ability  to  interoperate  with  the  DCS 
are  also  Included  since  they  carry,  or  can  carry,  DOD  communications  traffic. 

Other  potential  benefits  which  do  not  fit  readily  into  the  categories 
discussed  above  are  placed  in  Paragraph  5.5. 

Paragraph  5.6  looks  at  the  penalties  for  utilizing  precise  time.  The 
costs  in  terms  of  processor  and  overhead  channel  utilization  are  discussed. 

5. 1  Preliminary  Concepts 

In  this  paragraph  two  topics  are  introduced.  The  first  deals  with 
Coordinated  Universal  Time  (UTC).  The  second  describes  features  that  can  be 
included  in  a  timing  subsystem  to  improve  the  accuracy  with  which  it  can 
disseminate  precise  time  to  nodes  in  the  DCS. 

5.1.1  Coordinated  Universal  Time 

Throughout  this  study,  reference  to  precise  time  is  synonymous  with 
reference  to  Coordinated  Universal  Time. 

A  year  is  defined  as  the  time  it  takes  the  earth  to  orbit  the  sun. 

Prior  to  1972,  the  philosophy  of  precise  time  definition  was  to  attempt  to  control 
clocks  to  subdivide  the  year  into  an  agreed  upon  number  of  days,  hours,  minutes 
and  seconds.  Because  of  the  slow  orbital  motion  of  the  earth  and  the  difficulties 
of  identifying  when  the  earth  returned  to  a  given  point  in  space  using 
astronomical  observations  to  define  when  a  year  began  and  ended,  measurement 
uncertainties  limited  the  realization  of  accurate  ephemeris  time  to  about  0.05 
second  (50  milliseconds)  for  a  9-year  average.  The  development  of  the  Cesium  Beam 
Atomic  Clock  with  long  term  stabilities  In  the  range  of  10“H  or  better  led  to 
the  realization  that  an  error  in  such  a  device  would  be  in  the  range  of  500 
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microseconds  or  less  In  a  year  and  averaging  the  output  of  ten  to  twenty  atomic 
standards  could  push  stabilities  beyond  10-14. 

The  International  Radio  Consultative  Conmittee  (CCIR),  meeting  in 
Geneva  in  February  1971,  adopted  a  system  of  Universal  Timing  based  on  atomic 
clock  standards  with  provisions  for  periodic  coordination  of  time  standards  kept 
by  various  agencies  of  various  countries  throughout  the  world.  UTC  stands  for 
Universal  Time  Coordinated.  The  U.S.  Naval  Observatory  and  the  National  Bureau  of 
Standards  in  the  United  States  independently  maintain  equipment  to  generate  UTC 
and  these  time  bases  are  referred  to  as  UTC(USNO)  and  UTC(NBS)  respectively.  The 
Canadians  maintain  a  time  standard  in  Ottawa  referred  to  as  UTC(NRC)  and  the 
Germans  maintain  a  standard  In  Hamburg  referred  to  as  UTC(DHI).  All  of  these  plus 
many  other  standards  elsewhere  In  the  world  are  traceable  (periodically  compared 
with)  UTC(BIH)  which  is  maintained  by  the  Bureau  International  de  L'Heure  in  Paris 
which  processes  input  information  supplied  by  all  participating  organizations. 
Timing  information  from  any  of  these  organizations  is  traceable  to  UTC(BIH)  and  is 
usually  within  a  few  microseconds  of  it. 

The  time  difference  between  UTC(NBS)  and  UTC(USNO)  is  less  than  6 
microseconds.  Any  recognized  source  of  UTC  is  considered  to  be  precise  time. 

5.1.2  Relationship  of  Timing  Subsystem  Features  to  Precise  Time 

Of  the  timing  subsystem  features  described  in  Paragraph  2.2  the  first 
five  are  believed  to  offer  the  greatest  opportunities  to  improve  the  accuracy  with 
which  the  system  can  disseminate  precise  time.  They  are: 

1.  Directed  Control  (DC) 

2.  Double  Endedness  (DE) 

3.  Independence  of  Clock  Error  Measurement  and  Correction  (ICEMiC) 

4.  Phase  Reference  Combining  (PRC) 

5.  Self-Organization  (SO) 
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Not  one  of  these  features  is  absolutely  required  in  a  system  which  is 
to  disseminate  precise  time.  For  example,  a  Mutual  Synchronization  system  using 
single  endedness  can  be  referenced  to  precise  time  at  one  point  in  the  network. 

The  system  can  be  designed  to  be  stable  and  its  long  term  average  frequency  at 
every  mode  will  be  the  same.  The  precise  time  or  phase  offsets  at  all  nodes  in 
the  network  will  be  bounded  and  dependent  on  transmission  time  delays  and  varia¬ 
tions  in  those  delays.  If  some  links  in  that  Mutual  Sync  network  contain  satel¬ 
lite  links  the  phase  errors  may  well  be  measured  in  tenths  of  a  second,  but  being 
referenced  to  a  source  of  UTC,  those  errors  are  absolute  errors  relative  to 
precise  time  and  still  probably  of  sufficient  accuracy  to  time  simultaneous 
athletic  events  in  different  parts  of  the  world.  The  Mutual  Sync  system  thus 
described  does  not  contain  any  of  the  features  mentioned  above. 

The  advantages  of  these  features  are  that  each  one  contributes  to  the 
improvement  of  absolute  accuracy  of  precise  time  dissemination  through  the  timing 
and  synchronization  subsystem. 

Directed  Control,  described  in  Paragraph  2.2.1,  improves  the 
dissemination  of  precise  time  when  the  best  quality  path  (or  using  phase  reference 
combining,  the  best  set  of  paths)  from  the  source  of  precise  time  to  each  point  in 
the  network  is  chosen.  Usually,  though  not  necessarily  always,  the  path  or  paths 
chosen  will  be  the  shortest  paths  encompassing  the  least  transmission  time  delay. 
If  no  provision  is  made  to  dynamically  compensate  for  transmission  time  delay, 
nominal  (or  one  time  measured)  transmission  delays  can  be  used  to  make  better 
predictions  of  precise  time  over  the  designated  path  or  paths. 
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Double  Endedness  is  described  in  Paragraph  2.2.2  where  it  is  shown  that 
the  phase  (or  precise  time)  difference  between  the  clocks  at  both  ends  of  a  duplex 
link  is, 

=  (Ka  -  kb)/2  ♦  (0AB  -  Dba)/2. 

Ka  and  Kg  are  the  measurements  made  at  each  end  of  the  link.  DAg  and  DgA 
are  the  transmission  time  delay  in  each  direction  over  the  duplex  link.  If  trans¬ 
mission  time  delays  are  equal  in  both  directions  then  the  precise  time  error 
between  clocks  is  known.  Measurement  errors  in  KA  and  Kg  and  differences  in 
equality  of  transmission  time  delay  in  both  directions  both  contribute  to  errors 
in  4t.  However  both  of  these  errors  in  dt  will  be  very  much  smaller  than  coirmon 
mode  nominal  delay  and  common  mode  delay  variation.  Thus  Double  Endedness 
eliminates  a  major  portion  of  the  precise  time  error  contributed  by  transmission 
time  delay.  Also  note  that  when  a  duplex  link  is  instrumented  to  periodically 
make  these  Double  Endedness  measurements  dynamically  changing  corimon  mode 
transmission  time  delay  variations  at  rates  less  than  one  half  the  sampling  period 
are  cancelled  out.  This  is  particularly  important  in  duplex  links  through 
satellite  repeaters  which  have  a  daily  vertical  drift  imposing  a  significantly 
large  comnon  mode  change  in  transmission  time  delay. 

Independence  of  clock  error  measurement  and  correction  is  described  in 
Paragraph  2.2.3.  When  this  technique  is  used  throughout  a  network  having  directed 
control  and  Double  Endedness  all  nodes  in  the  network  have  control  loop  input 
errors  which  are  "relative  to  the  master"  with  the  measured  but  uncorrected  errors 
of  intervening  nodes  cancelled  out,  except  for  the  obvious  accumulation  of  errors 
in  the  intervening  measurements  and  differences  in  transmission  time  delay. 
Independence  of  clock  error  measurement  and  correction  is  not  compatible  with 
mutual  systems. 
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Phase  Reference  Combining  as  described  in  Paragraph  2.2.4,  is  an 
additional  refinement  on  the  above  described  features.  When  two  or  more  paths 
exist  between  a  node  and  the  precise  time  source  the  combining  of  this  information 
in  proportion  to  the  quality  of  the  information  should  further  improve  the 
expected  accuracy  of  the  timing  information.  For  the  phase  reference  combining 
feature  to  really  be  effective,  the  independence  of  clock  error  measurement  and 
correction  feature  should  also  be  included. 

The  last  four  features  discussed  dynamically  and  continuously  affect 
the  accuracy  of  precise  time  dissemination  while  the  Network  Configuration  is 
static.  Self-organization  and  self-reorganization  capability  is  related  to  the 
accuracy  of  precise  time  dissemination  at  a  node  following  some  types  of 
failures.  It  allows  a  highly  connected  network  to  alter  its  time  distribution 
organization  such  that  the  effect  of  most  failures  will  have  little  or  no  effect 
on  the  ability  to  continue  to  operate  over  different  combinations  of  paths  and/or 
through  different  nodes. 

In  addition  to  improving  the  accuracy  of  precise  time  these  features 
will  contribute  to  improved  reliability,  survivability  and  availability.  The 
information  transferred  and  the  measurements  and  calculations  required  for 
implementing  these  features  provide  many  possible  points  for  system  performance 
monitoring. 

The  ability  to  alarm  operations  and  maintenance  personnel  of  a  pending 
problem  that  might  require  their  personal  attention  while  there  is  still  time  to 
take  corrective  action  before  the  problem  becomes  serious  may  in  many  cases  make 
it  possible  to  make  necessary  corrections  without  disruption  of  service. 

These  features  may  be  included  in  various  logical  combinations  to 
instrument  timing  subsystem  synchronization  methods  such  as  Master-Slave,  Mutual 
Sync,  and  variations  thereof.  Any  of  these  systems  can  be  referenced  to  UTC 
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producing  a  system  which,  being  slaved  to  precise  time,  disseminates  precise  time 
to  an  absolute  time  accuracy  which  is  a  function  of  the  degree  to  which  each 
feature  is  included  and  refined. 

5.2  Potential  Benefits  of  the  Availability  of  Precise  Time  for  DCS  Network 

Synchronization 

There  are  several  benefits  to  DCS  communications  network  synchroniza¬ 
tion  when  precise  time  is  made  available  to  the  various  major  nodes  within  the 
network. 

•  Aids  Monitorability  and  Self-Reorganization 

•  Reduces  Phase  Disturbances  and  Improves  Frequency  Accuracy 

•  Improves  Bit  Synchronization  and  Signal-to-Noise  Performance 

•  Improves  accuracy  at  which  system  can  free  run  following  failure 

5.2.1  Aids  Monitorability  and  Self-Reorganization 

Monitorability  and  reorganization  are  integrally  tied  together  in  the 
sense  that  there  is  no  reason  to  monitor  system  performance  unless  it  will 
initiate  corrective  action  in  the  event  of  malfunction.  Granted  the  corrective 
action  may  be  automatic  (self-reorganization)  or  manual,  and,  if  manual,  can  be  in 
real  time  or  after  system  shutdown.  Certainly  self-reorganization  in  real  time 
cannot  be  implemented  without  at  least  as  high  a  degree  of  monitoring  and/or 
decision  capability  as  is  necessary  to  alarm  manual  corrective  action  in  real  time 
which  in  turn  requires  either  more  rapid  decision  making  and/or  more  complete 
monitorable  data  than  a  corrective  action  (or  design  change)  made  after  system 
shutdown. 

The  following  discussion  shows  that  the  availability  of  precise  time 
will  improve  monitorability  which  will  In  turn  improve  the  response  time  for 
self-reorganization.  The  following  list  itemizes  features  utilizing  precise  time 
which  aid  subsystem  monitorability  and  self-reorganization: 

•  Uniform  Time  Base  for  Algorithm  Accuracy  and  Simplification 

•  Subsystem  Health  Monitoring 
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Uniform  Time  Base  for  Algorithm  Accuracy  and  Simplification 

In  the  dissemination  of  precise  time  each  major  node  in  the  network  is 
being  disciplined  to  UTC(USNO).  This  provides  a  very  accurate  time  base  standard 
at  a  node. 

In  order  to  monitor  various  parameters  in  the  timing  subsystem,  it  is 
necessary  to  estimate  the  future  variations  or  change  at  a  particular  point  in 
time.  The  purpose  of  estimating  the  future  value  of  a  parameter  is  to  compare  it 
with  the  actual  value  that  occurs.  A  malfunction  decision  can  be  made  based  on 
the  maximum  deviation  between  the  estimated  value  and  the  actual  value. 

Two  of  the  advantages  of  using  the  features  discussed  in  Paragraph 
5.1.2  are  that  link  delay  variations  are  not  propagated  down  the  network  hierarchy 
and  the  nodal  clock  is  periodically  updated  and  disciplined  to  Coordinated 
Universal  Time.  These  factors  produce  a  more  uniform  time  scale.  Hence,  any 
estimating  algorithm  which  is  a  function  of  time  will  be  more  accurate. 

Assume  that  the  number  of  bits  in  the  buffer  of  an  incoming  data  link 
is  being  monitored.  It  is  known  that  over  an  increment  of  time.  At,  the  change  in 
the  number  of  bits  must  be  less  than  some  value  Ab.  If  the  limit  is  exceeded,  a 
warning  is  provided  to  an  operator.  Bits  are  being  clocked  into  the  buffer  by  the 
timing  obtained  from  the  incoming  data  stream.  Bits  are  being  read  out  by  the 
nodal  clock.  Since  there  are  normal  variations  in  the  incoming  bit  stream,  there 
is  an  expected  variation  in  the  number  of  bits  from  a  nominal  value  after  a  time 
interval  At  has  elapsed.  The  number  of  bits  can  be  modeled  by  the  following 
expression, 

J  =  b,t2  +  b,t  +  b„ 

2  10 

J  is  the  number  of  bits  in  the  buffer  at  some  time  t.  In  order  to  estimate  J,  the 
coefficients  bQ>  t>i,  and  b2  must  be  updated.  Techniques  employed  to  update 
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these  coefficients  are  those  found  in  polynomial  estimation  theory  and  state  space 

estimation  theory.  To  estimate  the  change  in  the  number  of  bits,  the  following 

equation  can  be  employed. 

a  N  r  212 

J  (T)  =  E  |w0(i)  +  W1(i)r+  W2(i )  T  ^  J  J(t  -  i  At) 

“o.  wl.  W2  are  found  by  standard  polynomial  filtering  techniques.  N  Is 
the  number  of  samples  for  which  the  buffer  size  J  is  observed.  T  is  the  time  in 

the  future  at  which  one  desires  to  determine  how  the  actual  JQ  deviates  from  the 

A  A  A 

predicted  J;  that  is,  |  J  -  bit$. 

If  the  time  scale  is  in  error,  then  by  using  differentials  one  obtains 
the  following  equation  which  depicts  the  error  that  is  produced. 

a  N  T  1 

AJ(T)  -  E  |^W1(i )  +  W2(i)T  AT  J(t  -  i  At) 

A T  represents  the  error  in  the  time  scale  measurement. 

The  following  points  should  be  noted: 

A 

•  The  error  AJ(  7")  in  the  buffer  estimate  is  dependent  on  the 
accuracy  A7"  of  the  time  scale.  Note  as  A T  approches  zero,  the 
inaccuracy  due  to  AT  approaches  zero. 

•  For  systems  which  have  a  strong  acceleration  coefficient  (W2(i) 

in  this  example)  such  as  satellite  systems,  the  situation  is  worse. 

•  The  further  away  the  estimate  T  is  made  from  the  present  time,  t, 

A 

the  greater  AJ  (T)  becomes. 

•  Higher  order  polynomials,  which  are  better  estimators,  produce  more 

A 

terms,  hence  tending  to  Increase  AJ( T  )  and  nullifying  the  extra 
terms.  Keeping  A T  small  reduces  this  problem. 

This  is  an  example  of  one  technique  for  monitoring.  However,  it  should 
provide  the  reader  with  some  appreciation  of  the  merits  of  utilizing  precise  time. 
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Subsystem  Health  Monitoring 

Timing  information  Is  periodically  transferred  between  nodes  in  an 
improved  TRD  system.  Since  each  node  possesses  computational  resources,  the 
techniques  for  computer-communications  health  monitoring  can  be  utilized.  These 
include  providing  a  software  monitor  to  determine  whether  the  proper  information 
has  been  received  at  a  node.  Thus,  the  monitor  at  a  node  has  to  determine  if  the 
link  is  delivering  timing  data  and  whether  it  is  correct.  The  latter  situation 
has  already  been  discussed.  The  former  situation  is  representative  of  a  technique 
in  which  dual  CPU's  in  a  computer-computer  communications  system  check  on  each 
other  to  determine  whether  they  are  still  functioning  properly.  When  it  is 
determined  that  a  failure  has  occurred,  the  system  reorganizes.  Precise  time  is 
beneficial  to  determining  when  a  period  of  time  has  elapsed  and  no  information  has 
been  received.  Thus,  precise  time  is  useful  in  making  a  prompt  decision  that  a 
failure  has  occurred. 

5.2.2  Reduces  Phase  Disturbances  and  Improves  Frequency  Accuracy 

Consider  a  single  duplex  link  between  a  master  frequency  source  and  a 
directly  connected  node.  In  a  conventional  master  slave  approach  the  input  to  the 
phase-locked  loop  of  the  controlled  node  will  contain; 

a)  Variations  due  to  drifts  and  Instabilities  of  the  master  clock,  plus 
b)  variations  due  to  transmission  time  delay  changes,  plus  c)  variations  due  to 
drifts  and  instabilities  of  the  clock  being  controlled.  If  the  master  clock  is  a 
source  of  UTC  then  the  variations  due  to  drifts  and  instabilities  of  the  master 
clock  will  be  as  small  as  can  presently  be  obtained.  This  assumes  that  the  USNO 
and  other  keepers  of  UTC  incorporate  every  practical  improvement  in  their 
standards.  Thus  the  first  contributor  to  variations  of  the  input  to  a  phase 
locked  loop  is  minimized  as  a  direct  result  of  being  slaved  to  UTC. 
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The  second  contribution  to  variations  (due  to  transmission  time  delay 
changes)  is  reduced  by  the  use  of  Double  Endedness  measurements.  When  there  are 
additional  nodes  between  the  master  clock  and  the  node  being  controlled, 
independence  of  error  measurement  and  correction  minimizes  absolute  time  errors. 
These  last  two  improvements  are  a  result  of  the  means  by  which  precise  time  is 
distributed  rather  than  the  fact  that  it  is  precise  time. 

The  variations  due  to  drift  and  instabilities  of  the  controlled  clock 
can  only  be  minimized  by  improving  the  quality  (and  cost)  of  the  clocks  at  each 
node. 

It  should  be  obvious  from  this  discussion  that  both  phase  and  frequency 
disturbances  will  be  reduced  through  application  of  these  techniques,  partly 
because  of  the  direct  availability  of  precise  time  and  partly  because  of  the  means 
by  which  that  time  is  disseminated. 

5.2.3  Improves  Bit  Synchronization  and  Signal  to  Noise  Performance 

It  has  been  explained  in  the  previous  section  that  the  phase 
disturbance  is  reduced  with  dissemination  of  precise  time.  The  rationale  was  that 
for  other  techniques  link  variations,  clock  instability  at  the  other  end  of  the 
communications  link,  and  perturbations  from  nodes  more  remote  than  one  link  away 
which  filtered  through  the  node  filter  at  the  other  end  of  the  link  all 
contributed  to  disturbances  at  the  local  node.  Furthermore,  by  distributing 
precise  time  from  a  stable  source,  there  is  a  tendency  to  fix  the  frequency 
throughout  the  network. 

Bit  synchronizers  are  incorporated  in  the  multiplexers  in  digital 
networks.  The  purpose  of  the  bit  synchronizer  is  to  recover  the  clock  rate  from 
digital  data  and  is  typically  instrumented  using  a  phase-locked  loop  to  track  the 
data  transitions.  A  phase-locked  loop  Is  a  tracking  filter  with  capabilities  to 
free  run  between  variable  Inputs.  When  the  source  of  the  frequency  to  be  tracked 
by  the  phase-locked  loop  is  highly  accurate  the  bandwidth  of  the  phase-locked 
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tracking  filter  can  be  made  narrower.  This  results  In  the  ability  to  track  the 
desired  frequency  in  a  much  lower  signal  to  noise  environment  than  would  be 
possible  if  the  frequency  source  were  less  stable  and  the  bandwidth  of  the 
phase- locked  loop  had  to  be  made  much  wider. 

5.2.4  Improves  Accuracy  at  Which  a  Node  Can  Free  Run  Following  Failure 

If  for  any  reason  a  node  in  the  network  loses  its  source  of 
synchronization  information  it  is  desirable  that  the  local  node  continue  to 
operate  in  a  free  running  or  independent  clock  mode  of  operation  until  the  source 
of  timing  can  be  established.  Assume  a  node  in  the  network  has  been  in  operation 

for  weeks  or  months  prior  to  a  timing  system  input  failure.  A  history  of  the 

systematic  errors  of  the  local  clock  can  be  accumulated  during  the  time  it  is 
controlled.  These  measurements  can  then  be  used  to  predict  the  necessary  control 
to  keep  the  clock  running  in  a  more  accurate  phase  relationship  with  the 
disconnected  system  for  longer  periods  of  time  when  the  source  of  timing 
information  is  lost  or  seriously  degraded.  The  longer  the  time  history  data  is 
collected  the  more  accurate  the  predictions  of  expected  systematic  error  should  be. 

The  length  of  time  over  which  the  increased  accuracy  can  be  useful 

primarily  depends  on  two  things;  first,  upon  the  quality  of  the  clock  used  at  the 

local  node;  and,  second,  upon  the  accuracy  of  the  clock  to  which  the  node  was 
controlled  prior  to  the  failure  which  caused  the  loss  of  control  input.  This 
accumulation  of  time  history  of  controlling  one  clock  from  another  clock  will 
produce  a  prediction  of  the  difference  between  the  two  clocks  in  future  time.  If 
the  master  source  oscillator  were  of  the  same  quality  as  the  oscillator  being 
controlled  predictions  would  be  based  on  the  relative  variations  between  clocks  of 
which  half  the  variation  would  be  in  the  source.  If,  on  the  other  hand,  the 
source  had  a  hypothetical  perfect  stability  over  any  time  interval  the  predictions 
would  only  involve  the  variations  in  the  controlled  clock,  and  in  this  example  the 
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variations  should  only  be  half  as  great.  Thus  precise  time  availability  during 


normal  slave  mode  operation  provides  the  ability  to  isolate  the  systematic  errors 
to  the  local  clock  and  to  compensate  for  them  more  accurately  during  a  failure 
which  causes  the  controlled  node  to  free  run. 

5.3  Potential  Benefits  of  the  Availability  of  Precise  Time  for 

intrasystem  Operability 

In  this  section  precise  time  is  discussed  with  respect  to  promoting 
intrasystem  interoperability.  This  Implies  synchronizing  equipment  at  the  various 
nodes  within  the  network.  The  effects  of  precise  time  on  networks  external  to  the 
DCS  are  discussed  in  the  next  section.  The  following  areas  are  analyzed  in  this 
section. 

5.3.1  Promotes  Efficient  Subsystem  Equipment  Synchronization 

There  are  equipment  functions  within  the  DCS  network  which  become  more 
efficient  when  precise  time  Is  utilized.  Two  of  these  functions  are  spread 
spectrum  equipment  synchronization  and  low  rate  data  communications  channel 
multiplexing. 

Spread  Spectrum  Equipment 

Direct  spread  spectrum  techniques  are  employed  in  modems  to  solve  the 
antijam,  low  probability  of  intercept,  ranging,  and  multiple  access  problems. 

These  signal  processing  techniques  will  be  used  in  the  DCS.  An  example  is  the 
impending  procurement  of  the  SHF  DA-TDMA  modem,  which  will  exploit  the  wideband 
communications  link  of  the  DSCS  satellites  to  achieve  unique  switchable  trunking 
capability  between  network  elements.  There  are  other  modems  In  existence  which 
require  and  utilize  direct  spread  spectrum  techniques. 

Characteristics  of  a  spread  spectrum  signal  are  that  the  transmitted 
signal  be  antipodal  (NRZ  data  multiplied  by  the  carrier  to  obtain  phase  shift 
keying  satisfies  this  condition)  and  have  its  frequency  spectrum  spread  by  a 
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pseudorandom  (pn)  code.  The  modulated  signal  can  be  regenerated  (despread)  If  the 
receiver  is  properly  synchronized.  The  following  three  items  must  be  known  to 
retrieve  the  signal. 

•  Code  Polynomial 

t  Code  Phase 

•  Code  Clock  Rate 

The  code  polynomial  of  the  pn  sequence  is  known  at  each  end  of  the 
link.  Higher  order  polynomials  produce  more  secure  codes.  For  code  division 
multiple  access,  each  station  would  have  s  unique  address  or  polynomial  assignment. 
In  applications  like  ranging,  the  code  would  probably  never  change.  However,  for 
anti jam  applications,  each  station  can  have  either  a  code  of  the  day  list,  or  code 
tap  settings  could  be  sent  as  data  over  the  link  or  an  auxiliary  link. 

The  code  clock  ate  is  generally  decided  upon  during  engineering 
design.  A  typical  chip  rate  being  employed  by  Harris  Corporation  in  their  large 
sophisticated  terminal  systems  Is  20  megachips  per  second  (Mc/s). 

Determining  the  code  phase  is  difficult.  The  stations  communicating 
must  start  their  codes  at  the  same  time  with  the  same  initial  loading  (plus 
propagation  delay).  Typically,  time  must  be  accurate  to  within  microseconds  for 
direct  pn  spread.  This  will  minimize  the  search  time.  An  error  of  +1  minute 
would  cause  an  uncertainty  of  +1.2  x  109  chips  in  a  20  Mc/s  system.  If  the 
error  is  10  microseconds,  this  would  cause  an  uncertainty  of  only  200  chips. 

Another  interesting  facet  of  utilizing  a  system  that  disseminates 
precise  time  is  that  the  phase  and  frequency  at  each  of  the  nodes  is  being 
disciplined  to  the  same  value  of  frequency  and  phase  throughout  the  network. 

Assume  that  some  other  technique  is  employed,  and  the  frequency  difference  between 
two  nodes  is  +1  x  10-U.  Then  for  a  20  Mc/s  code  rate,  the  maximum  search  time 
for  various  periods  are  given  in  Table  5.3.1. 
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Table  5.3.1.  Maximum  Time  to  Acquire  a  Spread  Spectrum  Signal 
When  Clock  Is  Updated  According  to  the  Given  Period 


Update 

Search 

Period 

Period 

1  Day 

1.4  Seconds 

1  Week 

10  Seconds 

1  Month 

42.86  Seconds 

6  Months 

4.29  Minutes 

A  very  small  frequency  discrepancy  was  chosen  for  this  example.  Note 
that  as  the  frequency  discrepancy  between  two  nodes  grows,  the  search  period  grows. 

Data  Communications  Channel  Multiplexing 

In  many  communications  systems  there  are  low  rate  data  streams  that  can 
be  multiplexed  together.  For  example,  teletype  information,  low  speed  data 
channels  that  carry  digital  data,  and  instrumentation  information  can  occur  at 
9600  bits  per  second  or  less.  Because  of  the  low  rate  involved,  these  streams  can 
be  interleaved  utilizing  the  nodal  clock.  Assume  the  nodal  clock  is  accurate  to 
+10  microseconds.  This  is  a  reasonable  accuracy  to  expect.  Since  a  bit  is 
received  every  104  microseconds,  four  data  streams  can  be  accurately  interleaved 
using  the  nodal  clock  for  timing.  This  simplifies  the  circuit  design  for  timing. 
With  precise  time  the  clocks  can  be  maintained  to  within  +10  microseconds.  In  the 
discussion  of  the  Loran  C  chains  in  Paragraph  5.4,  it  is  pointed  out  that  the 
design  goal  is  +2.5  microseconds. 

5.3.2  Reduce  System  Operational  Costs 

More  than  1200  cesium  beam  standards  are  in  use  by  the  DOD.  The  list 
of  users  will  continue  to  grow  with  the  acquisition  by  the  DOD  of  new  satellite 
systems,  upgrading  of  missile  warning  systems,  and  Implementation  of  new  and 
upgrading  of  old  navigation  systems.  Calibration  of  a  cesiint  standard  can  be 
performed  by  means  of  radio  dissemination  of  time  and  frequency,  a  portable  clock, 
and  other  methods  such  as  optical  pulsar  signals,  telephone  line  and  coax 
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transmission,  and  power  line  signals.  These  other  methods  are  the  least  accurate 
or  lend  themselves  only  to  special  situations.  The  first  two  methods  cited  above 
are  more  precise. 

One  objective  of  utilizing  a  system  to  distribute  time  is  to  have 
precise  time  available  to  a  broad  set  of  users.  Since  the  DCS  will  be  a  world  wide 
system,  any  synchronizing  technique  which  distributes  precise  time  will  produce 
cost  savings  with  respect  to  eliminating  the  need  for  transporting  atomic  clocks. 

A  graph  has  been  generated  in  Figure  5.3.2  which  depicts  the  cost  of 
transporting  cesium  atomic  clocks  to  perform  clock  corrections.  These  data  are 
based  on  travel  costs  for  two  men,  a  senior  engineer  and  a  technician.  Included 
in  the  costs  are  travel,  salary,  overhead,  and  general  and  acini ni strati ve  costs. 

The  costs  for  trips  greater  than  10  days  are  based  on  overseas  travel  costs. 

To  use  the  graph  consider  the  following  example.  If  ten  clocks/year 
require  5-day  trips  and  ten  clocks/year  require  10-day  trips,  the  total  cost  is 
estimated  as  $32, 700/year.  For  100  clocks  requiring  3-day  trips,  the  cost  is 
$100, 000 /year. 

As  the  above  figures  indicate,  the  savings  when  distributing  precise 
time  via  a  network  timing  scheme  can  be  substantial. 

5.4  Potential  Benefits  of  the  Availability  of  Precise  Time 

for  POD  Intersystems  Applications 

Intersystem  interoperability  is  a  major  driving  force  for  utilizing 
precise  time  in  the  DCS  conmuni cations  network.  This  is  discussed  under  the 
following  listed  subheadings. 

•  Compliance  With  Federal  Standards 

•  Facilitates  System  Interoperability 

•  Facilitates  Experiments  Employing  Precise  Time 

•  Improves  the  Maintainability,  Reliability,  and  Capability  of 
Navigation  Systems 
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Figure  5.3.2.  Cost  of  Portable  Atomic  Clock  Calibration 
Versus  Number  of  Days  in  the  Field 
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Compliance  With  Federal  Standards 

A  major  reason  for  having  the  DCS  referenced  to  precise  time  is  to 
comply  with  Federal  Standard  1002.  This  standard  was  issued  by  the  General 
Services  Administration  pursuant  to  the  amended  Federal  Property  and 
Administrative  Services  Act  of  1949.  The  purpose  of  this  standard  is  to 
facilitate  interoperability  between  telecommunication  facilities  and  systems  of 
the  Federal  Government.  It  is  stated  in  this  standard  that  it  shall  be  used  by 
Federal  agencies  where  interoperability  between  Federal  Government 
telecommunication  facilities  and  systems  are  dependent  on  time  and  frequency 
reference  information. 

The  general  requirement  is  that  time  and  frequency  information  utilized 
in  applicable  Federal  Government  telecommunication  facilities  and  systems  shall  be 
referenced  to  the  existing  standards  of  time  and  frequency  maintained  by  United 
States  Naval  Observatory,  UTC(USNO),  or  the  National  Bureau  of  Standards, 

UTC(NBS).  Coordinated  values  of  UTC  are  used  to  obtain  standard  values  of  time 
and  frequency.  With  respect  to  accuracy  the  standard  states  that  time  and 
frequency  reference  information  with  respect  to  UTC  at  the  USNO  or  NBS  shall  be 
commensurate  with  the  individual  systen  design  and  interface  requirements. 

Presently  DCA  is  committed  to  provide  DSCS  terminals  with  precise 
clocks  that  are  disciplined  to  UTC(USNO)  via  time  transfer  over  satellite 
communications  links. 

5.4.2  System  Interoperability 

In  this  paragraph  network  timing  subsystem  interoperabi 1 Ity  is 
considered  with  respect  to  Interfacing  between  the  DCS  and  other  communications 
and  navigation  networks.  Examples  of  communications  networks  are  DATRAN  and 
AT&T's  Dataphone  Digital  Service.  Navigation  systems  are  included  in  this 
paragraph  because  they  utilize  UTC(USNO)  and  could  become  either  a  source  of  UTC 
or  a  user  of  UTC  distributed  by  another  network. 
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There  are  several  reasons  for  desiring  system  interoperability.  These 


r~ 

I 


surmiarized  as  follows: 

Reduce  Interfacing  Requirements  for  Increased  World  Coverage 
More  Timing  Subsystem  Redundancy 
Reduce  Traffic  Interruptions 


reasons  can  be 

|i  i 

!  i  : 


•  Simplify  Intersystem  Computations 

The  theme  of  this  paragraph  is  that  timing  subsystems  which  employ  and  distribute 
UTC  would  be  more  likely  to  interface  with  each  other  effectively.  Given  that  two 
systems  use  the  same  source  of  precise  time,  and  that  precise  time  is  distributed 
to  the  gateway  nodes  of  each  system.  The  absolute  time  (phase)  difference  between 
the  two  networks  at  the  gateway  nodes  would  be  bounded.  Therefore,  the 
probability  of  buffer  overflow  and  the  need  to  reset  buffers  is  minimal.  This 
philosophy  is  in  keeping  with  DCA's  desire  for  reduction  of  communications  traffic 
interrupti ons. 

Reduce  Interfacing  Requirements  for  Increased  World  Coverage 

Several  commercial  digital  communications  networks  are  being 
implemented.  These  include  the  following: 

•  DATRAN 

•  Canadian  Dataroute 

•  AT&T's  DDS 

•  Western  Union 

•  MCI 

Digital  circuits  are  expected  to  be  leased  from  conmercial  carriers  for 
the  CONUS  and  transoceanic  parts  of  the  DCS.  Government  facilities  are  being 
considered  overseas  for  use  in  the  future  digital  DCS.  In  addition,  the  Defense 
Satellite  Communications  System  (DSCS)  will  be  part  of  the  PCS.  The  DOD  policy  is 
for  the  US NO  to  coordinate  frequency  and  timing  throughout  the  military  services. 
In  fact,  clocks  at  DSCS  terminals  are  being  coordinated  with  UTC(USNO). 
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Comnercial  networks  will  more  than  likely  be  coordinated  to  UTC(NBS)  in 
the  future,  which  is  traceable  to  UTC(USNO).  In  fact,  the  major  broadcast 
networks  disseminate  NBS  standard  frequencies.  The  direct  difference  between 
UTC(NBS)  and  UTC(USNO)  is  less  than  6  microseconds. 

Most  of  the  commercial  networks  cited  above  employ  a  master-slave 
approach.  If  a  master-slave  system  has  the  features  of  Double  Endedness  and 
independence  of  clock  error  measurement  and  correction,  it  can  distribute  precise 
time  to  its  various  nodes.  This  would  facilitate  interfacing  the  DCS  to 
commercial  networks;  hence,  wider  world  communications  coverage.  In  addition,  the 
limited  phase  difference  (<6  microseconds)  between  UTC(USNO)  and  UTC(NBS)  can  be 
handled  easily.  It  is  the  difference  in  clock  frequencies  which  contributes  to 
bit  slips.  However,  with  the  same  frequency  at  each  node  the  absolute  time 
(phase)  difference  is  bounded  and  buffers  of  practical  size  will  not  overflow. 

More  T iming  Subsystem  Redundancy 

This  attribute  is  fairly  obvious.  As  more  networks  employing  UTC  are 
interconnected  the  probability  of  having  precise  time  available  to  the  DCS  network 
during  periods  of  upheaval  is  increased.  Timing  can  be  supplied  to  a  node  or 
section  of  a  network  if  the  node  or  section  of  the  DCS  network  gets  cut  off  from 
the  master.  On  the  other  hand,  the  DCS  could  distribute  timing  to  interfacing 
fragmented  networks. 

As  the  discussion  in  Paragraph  5.4.3  will  point  out,  navigation  systems 
are  capable  of  disseminating  UTC(USNO).  Interfacing  these  systems  with  the  DCS, 
which  has  precise  time  traceable  to  the  Naval  Observatory,  would  be  advantageous 
with  respect  to  improving  the  survivability  of  the  DCS.  On  the  other  hand,  the 
DCS  could  be  utilized  to  monitor  the  timing  accuracy  of  navigation  systems. 
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Reduce  Traffic  Interruptions 

Traffic  interruptions  are  reduced  when  the  probability  of  a  bit  slip  is 
decreased  following  either  the  failure  of  a  node  in  the  network  or  destruction  of 
parts  of  the  network.  The  concept  that  is  important  here  is  that  each  node  is 
being  disciplined  to  precise  time.  Before  pursuing  this  further,  consider  the 
following.  Assume  that  the  frequency  difference  between  two  nodes  is  modeled  as  a 
step,  ramp,  or  sinusoidal  function: 

Step  Frequency  Difference: 

Fd  s  F1  -  F2  0  <  t  <  T 

F1  =  Frequency  at  Node  1 

F2  =  Frequency  at  Node  2 

Ramp  Frequency  Difference: 

Fd  =  (A0/T0)t  0  <  t  <  T 

Ao  3  Difference  in  frequency 
between  Nodes  1  and  2 
at  time  TQ 

Sinusoidal  Frequency  Difference: 

Fd  =  B0  sin  (  7i t/T0)  0<t  <T 

B0  =  Maximum  difference  between 

frequencies  at  Nodes  1  and  2 
during  the  period  2TQ> 

The  required  buffer  storage,  Q,  is  equal  to  twice  the  quantity  of  bits 
contained  in  a  time  duration  equal  to  the  timing  error,  Ey.  jhe  timing  error 
and  buffer  storage  are  given  by  the  following  expressions. 
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Step  Timing  Error: 

Q  a  2FrET  ,  2Fr(Fd/Fl)T 

pr  »  Incoming  bit  stream  rate 

Ramp  Timing  Error: 

Q  -  2FrEy  =  Fr(Ao/T0Fl)T2 

Sinusoidal  Timing  Error: 

Q  =  2FrET  *  4Fr(B0To/7rFi) 

From  these  equations  it  is  observed  that  to  minimize  the  possibility  of 
buffer  overflow  due  to  perturbations  on  the  link,  the  quantities  A0>  b0,  and 

should  be  made  as  small  as  possible.  Q  =  0,  when  Fj  -  F2  =  0.  The 
dissemination  of  UTC  precise  time  via  the  features  of  Double  Endedness  and 
independence  of  clock  error  measurement  and  correction  results  in  reducing  these 
quantities  to  zero. 

Simplify  Intersystem  Computations 

In  satellite  and  navigational  systems  there  are  many  mathematical 
equations  involving  time  which  have  to  be  evaluated  to  determine  location. 
Employing  a  common  time  base  facilitates  comparison  and  usability  in  other 
networks.  That  is,  calculations  made  in  one  network  are  more  readily  transferable 
to  another  network,  if  both  networks  employ  the  same  time  base. 

5.4.3  Improve  the  Maintainability,  Reliability  and  Capability 

of  Navigation  Systems 

There  are  several  navigation  systems  which  either  have  been  implemented 
or  are  being  implemented  that  will  utilize  precise  time  and  time  Interval 
standards  that  are  referenced  to  Coordinated  Universal  Time.  For  these  systems 
this  precise  time  reference  is  generally  disseminated  by  the  United  States  Naval 
Observatory.  The  following  navigation  systems  presently  use  or  will  use  precise 
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t  OMEGA 

•  LORAN  C 

•  TRANSIT 

•  NAVSTAR  Global  Positioning  System  (GPS) 

Each  of  the  systems  will  have  the  capability  to  interface  with  the 
DCS.  In  order  to  illustrate  how  the  DCS  referenced  to  precise  time  can  promote 
improved  reliability  and  capability  in  an  external  navigation  system,  a 
description  of  the  mission  and  requirements  for  navigation  systems  would  be 
useful.  The  OMEGA  system  will  be  discussed  first;  it  illustrates  the  requirements 
for  accuracy. 

OMEGA 

OMEGA  is  a  long-range,  all-weather  radio  navigation  system  consisting 
of  eight  transmitter  stations  strategically  located  around  the  world.  Each 
transmitter  generates  continuous  wave,  phase-locked,  very  low  frequency  (VLF) 
signals  between  10.2  and  13.6  kHz.  OMEGA  position  errors  are  typically  in  the 
order  of  1  to  2  nautical  miles  RMS.  VLF  signals  propagate  in  a  natural  waveguide 
between  the  earth's  surface  and  the  ionosphere,  and  maintain  a  nearly  linear 
relationship  between  signal  phase  and  distance  from  each  transmitter.  Phase 
difference  measurements  from  pairs  of  transmitters  define  earth-referenced 
hyperbolic  1 ines-of-position  (LOP's)  which  are  used  for  position  determination. 

OMEGA  utilizes  a  phase  differencing  technique  for  positioning.  Use  of 
phase  differences  for  OMEGA  navigation  requires  precise  phase  synchronization  of 
signals  from  all  OMEGA  transmitters.  (A  6-microsecond  synchronization  offset 
between  two  transmitters  can  result  in  a  1  nautical  mile  position  error.)  The 
phase  of  each  OMEGA  transmitter  is  controlled  by  its  own  on-line  cesium  clock  (and 
several  backup  clocks).  To  prevent  uncontrolled  time  offset  growth  in  the  OMEGA 
system,  internal  synchronization  is  accomplished  by  periodically  adjusting  the 
epoch  (i.e.,  time  or  phase)  of  each  OMEGA  transmitter  to  the  average  epoch  of  all 
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transmitters  (Mean  OMEGA  System  Time).  External  synchronization,  which  is  not 
necessary  for  navigation,  but  is  for  time  dissemination,  is  established  by 
maintaining  Mean  OMEGA  System  Time  at  a  known  constant  offset  from  l)TC  as 
maintained  by  the  USNO.  Transmitter  phase  (epoch)  adjustments  are  computed  by 
processing  both  Internal  and  external  timing  measurements.  Internal  measurements 
indicate  the  relative  timing  offsets  between  pairs  of  OMEGA  transmitters. 

External  measurements  indicate  the  time  offsets  between  individual  transmitters 
and  UTC(USNO) . 

LORAN-C 

LORAN-C  is  a  navigation  system  which  provides  precise  position  for 
ships,  submarines,  and  aircraft.  This  system  uses  an  LF  100  kHz  carrier  frequency 
and  pulse  transmission  to  form  hyperbolic  lines  of  position.  The  U.  S.  Coast 
Guard  now  operates  and  manages  the  system  and  was  made  the  agent  of  the  Naval 
Observatory  for  dissemination  of  precise  time.  The  basic  LORAN-C  unit  is  a  chain, 
consisting  of  a  master  station  and  two  or  more  slave  stations  located  within 
groundwave  range  of  the  master  transmitter.  There  are  now  eight  chains  in 
operation  worldwide,  which,  together  with  slave  stations,  comprise  a  total  of 
about  34  stations. 

The  LORAN-C  system  must  be  synchronized  to  within  +5  microseconds  of 
UTC  (USNO).  Each  Loran-C  chain  is  to  be  synchronized  with  each  other  LORAN-C 
chain  within  +5  microseconds.  The  Coast  Guard  intends  to  comply  with  both 
requirements  by  maintaining  each  LORAN-C  chain  within  +2.5  microseconds  of 
UTC(USNO).  To  maintain  the  desired  tolerances  the  joint  Chiefs  of  Staff's  master 
navigation  plan  calls  for  monitoring  station  accuracy  via  the  DSCS  and  also  to 
provide  communications  to  feed  back  Instructions  to  the  monitored  station  for 
d  sciplining  its  clock.  With  a  synchronization  subsystem  that  disseminates 
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precise  time  this  can  be  easily  facilitated.  Since  the  DSCS  is  to  be  part  of  the 
DCS,  precise  time  will  be  available  for  monitoring  and  disciplining  the  clocks  in 
the  LORAN-C  network. 

TRANSIT 

This  Naval  navigation  satellite  system  is  primarily  a  means  for 
determining  accurate  navigation  fixes  through  Doppler  measurements  of  accurate 
radio  signals  transmitted  from  low-altitude  moving  satellites.  The  TRANSIT  system 
became  operational  in  1964.  It  consists  of  five  satellites  with  onboard  clocks 
moving  in  nearly  circular  polar  orbit. 

A  fairly  elaborate  correction  network  exists  for  the  TRANSIT  system. 

It  consists  of  four  tracking  stations  in  Hawaii,  California,  Minnesota,  and  Maine, 
a  computing  center,  a  time  reference  link  to  the  USNO,  and  two  ground  satellite 
conmunication  sites  or  injection  stations. 

NAVSTAR  Global  Positioning  System  (GPS) 

The  GPS  now  under  development  is  expected  to  provide  the  ability  to 
measure  geographic  position  in  three  dimensions  to  within  10  meters.  This  will 
require  that  GPS  System  Timing  be  coordinated  to  better  than  100  nanoseconds. 

This  system  will  employ  24  satellites  for  continuous  navigation  assistance.  By 
the  early  1980's  two  dimensional  services  will  be  available  across  the  globe  while 
the  system  is  being  deployed  for  full  services  planned  by  1984.  By  comparing  the 
time  when  a  signal  is  transmitted  to  the  time  when  it  is  received,  distance  from  a 
satellite  is  determined.  Three  such  distance  spheres  intersect  giving  a  navigator 
his  position.  Accurate  measurement  requires  precise  knowledge  of  the  satellites' 
positions  and  transfer  time  to  within  100  nanoseconds  rms  error.  In  addition  to 
broadcasting  position  and  time  of  transmittal,  the  satellite  updates  the  user's 
Inexpensive  crystal  clocks.  Satellite  atomic  clocks  are  synchronized  through  a 
master  control  station  to  be  initially  located  at  Vandenberg  AFB.  Frequency 
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accuracy  is  2  x  10-10.  Updating  is  done  every  12  seconds  to  maintain  time 
within  a  minimum  of  10-12  seconds  for  the  Cesium  clocks. 

5. 4. 3.1  Improve  Navigation  System  Capability 

From  the  discussion  above  it  is  apparent  that  accuracy  within  a  system 
depends  on  how  well  the  transmitters  within  a  chain  are  synchronized  to  each 
other.  This  implies  that  accuracy  is  independent  of  external  timing.  However, 
the  navigational  accuracy  for  an  entity  which  is  deriving  its  position  utilizing 
transmitters  in  two  separate  chains,  depends  on  both  chains  having  the  same  time. 
Furthermore,  if  precise  time  UTC(USNO)  is  distributed  throughout  the  DCS,  it  will 
be  in  close  proximity  to  the  chains.  This  will  facilitate  distributing  precise 
time  to  the  chains  and  result  in  improved  accuracy.  Another  aspect  is  that  more 
chains  can  be  constructed  without  incurring  the  problem  of  updating  many  locations 
by  physically  transporting  atomic  clocks.  Hence,  a  larger  range  of  coverage  can 
occur  more  conveniently. 

5. 4. 3. 2  Improve  Navigational  System  Maintainability 

As  alluded  to  in  the  above  discussion  on  accuracy,  reliability  results 
from  referencing  the  navigation  system  to  the  DCS.  If  the  DCS  makes  precise  time 
available,  the  navigation  system  can  be  updated  conveniently.  If  a  failure  occurs 
in  the  navigation  system,  precise  time  is  available  for  disciplining  the  clock. 
This  reduces  the  system  downtime. 

5.5  Other  Potential  Benefits 

The  benefits  discussed  below  are  important  in  their  own  right. 
Therefore,  they  are  discussed  here  for  completeness. 

5.5.1  Facilitate  Precise  Time  Experiments 

There  are  many  experiments  being  carried  out  by  the  scientific 
community  which  involve  or  utilize  precise  time  and  time  interval.  The 
availability  of  precise  time  at  a  nearby  location  facilitates  experiments. 

Methods,  the  complexity  of  which  are  a  function  of  the  distance  to  the  source. 
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have  to  be  devised  for  transferring  precise  time  to  the  experimental  location.  By 
distributing  precise  time  throughout  a  large  network,  the  availability  to  a  wider 
array  of  scientific  users  is  possible. 

5.5.2  Provides  for  Future  Network  Requirements 

By  Implementing  a  system  that  distributes  precise  time,  future 
requirements  for  precise  time  will  automatically  be  taken  into  account.  That  is, 
unforeseen  future  requirements  for  precise  time  will  have  more  success  in  being 
implemented  if  precise  time  is  part  of  the  initial  network  philosophy. 

5.5.3  Improve  Survivability 

It  is  clear  from  the  previous  paragraphs  that  network  survivability 
would  be  greater  because  of  the  distribution  of  precise  time  within  the  network 
and  throughout  other  DOD  nets.  This  is  reiterated  here  to  stress  the  fact  that 
the  increased  use  and  distribution  of  precise  time  can  only  have  the  effect  of 
minimizing  bit  slips  and  increasing  system  stability  which  improves  the  chances  of 
survivability  under  stress. 

5.6  Penalties  for  Utilizing  Precise  Time 

Several  penalties  occur  when  disseminating  precise  time.  The  major 
disadvantage  is  that  additional  resources  are  needed.  Double  Endedness, 
Independence  of  Clock  Error  Measurement  and  Correction*  Phase  Reference  Combining 
and  Self-Reorganization  all  require  the  transfer  of  information  between  nodes  in 
an  overhead  channel  (or  dedicated  communications  channel).  A  digital  processor, 
and  additional  logic  are  needed. 

Section  4  of  this  report  provides  detailed  paper  design  and  cost 
Information  on  the  implementation  of  the  various  combinations  of  features.  The 
conclusions  are  that  the  implementation  of  all  the  features  Included  in  the 
Improved  Time  Reference  Distribution  System  will  not  Increase  the  cost  of  the 
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synchronization  subsystem  very  significantly.  Section  3  shows  that  all  the 
features  are  useful  in  improving  the  accuracy  of  the  timing  at  all  nodes  in  the 
system. 

Since  information  is  being  transmitted  between  nodes,  channel  resources 
are  needed.  The  equipment  being  developed  will  contain  system  control  (SYSCON) 
subchannels.  Each  signaling  and  control  channel  shall  provide,  in  addition  to  the 
framing  and  signaling  sub-channels,  the  ability  to  accommodate  two  full  duplex 
system  control  (SYSCON)  sub-channels.  These  subchannels  have  a  full  duplex  data 
rate  of  4  kilobits  per  second.  These  channels  will  carry  system  messages  and  will 
accommodate  the  timing  traffic.  From  the  discussion  in  Paragraph  3.3.7  and  the 
results  presented  in  Table  3. 3. 7. 5,  precise  time  can  be  implemented  with  an 
overhead  bit  rate  of  approximately  250  bits  per  second  (b/s).  The  Double  Ended 
feature  is  represented  by  63  b/s,  and  the  independence  of  error  measurement  and 
correction  is  represented  by  an  additional  63  b/s.  This  represents  less  than  2 
percent  of  the  channel  overhead,  but  is  about  0.001  percent  of  a  26  Mb/s  total 
trunk  rate. 

A  leap  second  has  to  be  taken  into  account  each  year.  The  reason  for 
this  is  that  the  atomic  clock  rate  is  not  exactly  commensurate  with  the  length  of 
the  day.  There  may  be  special  situations  where  a  "minute"  contains  61  (or  59) 
seconds  instead  of  the  conventional  60  seconds.  This  will  not  occur  more  often 
than  once  a  year. 

When  precise  time  distribution  is  implemented,  the  majority  of  the 
determental  effects  of  transmission  time  delay  is  eliminated  except  for  a  delso' 
asymmetry  term  (Dafj-Djja)/2.  Factors  which  affect  the  magnitude  of  this  term 
are  the  differential  transmit  time  between  the  two  nodes,  and  equipment 
del  ay/ temperature  variations. 

Another  penalty  is  that  additional  equipment  and  procedures  might  be 
necessary  to  transmit  precise  time  to  the  OCS.  Oust  what  equipment  and  procedures 

are  necessary  were  beyond  the  scope  of  this  study. 
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6.0  SURVEY  OF  IMPLEMENTATION  AND  OPERATIONAL  EXPERIENCE  WITH 

SYNCHRONIZATION  AND  TIMING  SUBSYSTEMS  OF  CURRENTLY  OPERATING  DIGITAL 

NETWORKS. 

6.1  Introduction 

There  are  (were)  several  wideband  digital  communications  networks 
operating  which  incorporate  synchronization  systems  similar  to  those  described  in 
Section  2.0  of  this  report.  Most  of  these  networks  were  originally  conceived  and 
developed  in  the  late  sixties  and  early  seventies,  and  initially  deployed  in  the 
early  seventies.  This  section  of  the  report  describes  the  results  of  a  survey 
that  has  been  conducted  on  a  number  of  these  systems.  The  survey  attempted  to 
learn  of  practical  aspects  and  experiences  in  the  design,  deployment,  and 
operation  of  these  networks. 

Three  specific  systems  surveyed  are  the  switched  digital  network 
developed  by  the  Data  Transmission  Company  and  now  operated  by  SP  Communications 
(hereafter  called  the  Datran  System);  the  digital  network  deployed  and  operated  by 
the  Computer  Communications  Group  of  the  Trans-Canada  Telephone  System  (Dataroute) 
and  a  digital  system  between  Cincinnati  and  Atlanta,  developed  and  deployed  by  the 
Western  Union  Telegraph  Co.  (hereafter  called  the  WU  System).  With  the  exception 
of  the  WU  System,  these  networks  are  in  service  today  carrying  digital  traffic. 

The  Western  Union  system  was  put  into  operation  in  late  1971  and  field  trials 
conducted  for  several  months.  The  system  was  never  put  into  service,  and  has 
since  been  dismantled. 

In  the  following  pages.  Paragraphs  6.2  describes  the  Western  Union 
system  and  a  discussion  of  the  planning /deployment  experiences  related  to  it. 
Paragraphs  6.3  and  6.4  relate  to  the  Dataroute  and  Datran  systems  respectively. 
Paragraph  6.5  contains  information  obtained  about  the  Bell  System's  engineering  of 
Timing  and  Synchronization  (T/S)  for  the  No.  4ESS  network.  Paragraph  6.6 
concludes  this  section  with  a  suninary  of  the  other  subsections  and  with  a  general 
discussion  comparing  the  considerations  for  the  commerical  networks  described  with 
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the  T/S  considerations  for  a  strategic  military  network  such  as  the  DCS.  It  is 
pointed  out  there  that  many  of  the  features  necessary  to  be  considered  in  a 
military  system  such  as  the  DCS  are  not  factors  to  be  considered  in  a  commercial 
system;  or,  as  in  the  case  of  survivability,  it  may  be  of  interest  to  a  commercial 
network,  but  it  is  not  a  strong  enough  factor  to  dominate  design  trade-offs. 

Thus,  while  the  engineering,  planning  and  design  of  the  commercial  networks,  along 
with  the  operational  experiences,  are  of  interest,  it  must  be  noted  that  the  two 
applications  have  distinctions  that  can  heavily  sway  design  trade-offs  and 
decision  in  different  ways. 

In  the  first  three  subsections  that  follow,  a  brief  description  of  each 
of  the  major  subsystems  of  the  three  specific  networks  is  given.  This  then  is 
followed  by  a  discussion  of  a  set  of  questions  which  were  posed  to  personnel 
associated  with  each  of  the  networks  in  the  development  and  early  operational 
phases.  These  questions  were  posed  during  personal  meetings  with  the  personnel 
for  each  of  the  systems  in  questions,  with  the  exception  of  the  Datran  network. 

For  the  Datran  system,  the  information  was  drawn  from  records  and  recollections  of 
the  author  of  this  section  of  the  report.  Said  author  was  in  the  employ  of  Data 
Transmission  Company  during  the  development  of  the  system.  Because  of  this 
association,  the  Datran  information  available  and  included  herein  is  more  detailed 
than  that  for  the  other  networks.  An  attempt  has  been  made  to  avoid  having  the 
extra  detail  give  an  appearance  of  advocacy  of  the  Datran  approach  and  network. 
Whether  or  not  the  attempt  is  successful  must  be  left  to  the  reader's  judgement. 

Original  plans  were  for  this  survey  to  include  a  section  on  the  private 
line  Digital  Data  System  (DOS)  deployed  by  American  Telephone  and  Telegraph. 
However,  due  to  pending  litigation,  AT&T  personnel  were  not  able  to  provide 
information  on  DOS.  An  interview  was  substituted  which  covered  Bell's  planning 
for  timing  and  synchronization  of  the  network  of  No.  4ESS  switching  centers 
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currently  being  deployed.  (This  network  is  known  as  the  Switched  Digital  Network 
-  SDN.)  This  interview  ranged  over  a  spectrum  of  activities  from  early  planning 
in  the  60's  to  current  plans.  Paragraph  6.5  below  contains  information  from  this 
interview.  Very  little  in  the  way  of  operational  experience  is  yet  available  so 
that  the  information  presented  is  almost  entirely  of  an  engineering  planning 
nature. 

The  questions  posed  to  the  technical  personnel  associated  with  the 
three  digital  systems  (WU,  Datran,  Dataroute)  were  as  follows: 

A.  Fundamental  Reasons  for  Choosing  Timing  and  Synchronization  (T/S)  As 
Implemented. 

1.  What  were  criteria  used  for  selecting  the  T/S  system? 

2.  Were  other  type  T/S  systems  considered? 

3.  On  what  basis  was  final  choice  made  -  performance  analysis;  cost 
analysis;  0/M  analysis;  simulation;  other? 

4.  Is  system  reference  to  UTC  or  other  standard  necessary  and/or 
desirable? 

5.  How  was  compatibility  with  other  systems  (international,  military, 
other  common  carriers,  etc.)  a  factor  in  the  system 
selection/design? 

B.  Theoretical  Design  of  Chosen  T/S  System 

1.  What  were  the  main  criteria  in  specifying  the  T/S  system? 

Did  it  have  to  "marry"  existing  hardware,  etc.? 

2.  Specifications  of  Hardware 

(a)  Stability  -  long/short  term 

(b)  Reliability 

(c)  Jitter  reduction 

(d)  Cost  goals 

(e)  0/M  aspects 


4077A 

3.  What  were  driving  criteria  on  timing  distribution  waveforms, 
interfaces,  etc.  "downstream"  from  clocks,  i.e.,  between  MUX's, 
from  MUX's  to  user,  etc.? 

Practical  Design/Engineering  Considerations 

1.  Were  any  unusual  obstacles  met  in  the  design/development  of 
hardware? 

2.  Were  specifications  as  finally  met  tighter  or  looser  than  original 
specs? 

Initial  Field  Deployment,  Testing,  O&M 

1.  In  the  field  trials  and  early  operation  of  the  network,  were  any 
unusual  problems  encountered? 

(a)  Were  craftsmen  able  to  install,  test,  troubleshoot,  or  was 
system  so  foreign  that  engineers  were  needed? 

(b)  Was  elaborate/non-routine  test  equipment  needed? 

2.  Are  there  any  changes  to  system  design  that,  in  retrospect,  would 
enhance  or  otherwise  favorably  influence  initial  field  deployment 
and  testing? 

System  Operations  Experience 

1.  Has  system  operated  as  expected?  If  not,  what  has  arisen  that  was 
unexpected? 

2.  Is  performance  adequate  in  retrospect? 

(3)  Is  reliability  adequate  in  retrospect?  Is  redundancy  used? 

(4)  Is  maintainability  adequate  in  restrospect?  What  have  been  MTTR 
experiences? 

(5)  Is  flexibility  for  growth/change  acceptable  in  retrospect?  Are  any 
future  requirements  now  foreseen  that  would  have  influenced  design? 

Any  Other  Information  Useful  to  U.S.  Government/DOD  in  Planning  Defense 
Communications  System  (DCS)? 
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6 .2  The  Western  Union  Digital  Network 

The  Western  Union  Telegraph  Company  designed  and  deployed  a  digital 
system  that  has  been  included  in  this  survey.  This  system  was  installed  between 
Cincinnati,  Ohio  and  Atlanta,  Georgia  in  1971  and  was  operated  as  a  test-bed  until 
it  was  dismantled  in  late  1973  or  early  1974.  The  system  utilized  line-of-sight 
microwave  as  the  basic  trunking  medium  and  provided  point-to-point  data  services. 
The  system,  while  only  deployed  over  a  limited  area  (Cincinnati  to  Atlanta)  was 
conceived  and  designed  to  be  eventually  expanded  on  a  nationwide  basis. 

A  technical  description  of  the  equipments  used  in  the  Western  Union 
network  follows. 

Western  Union  Radio  System 

Although  other  transmission  facilities  were  eventually  planned  to  be 
used,  the  Western  Union  network  from  Atlanta  to  Cincinnati  was  first  designed  to 
use  an  existing  960  channel  FDM-FM  (analog)  radio  system  as  the  basic  trunking 
medium.  This  radio  system  traverses  ten  microwave  hops  between  Cincinnati  and 
Atlanta.  Nine  repeater  stations  and  the  two  end  terminals  are  included.  Eight  of 
the  repeater  stations  are  heterodyne  repeaters  in  which  the  received  signal  is 
frequency  translated  from  the  6  GHz  received  RF  carrier  to  70  MHz,  amplified,  and 
retranslated  to  another  6  GHz  RF  carrier  for  transmission.  In  the  other  (ninth) 
repeater,  the  signal  is  translated  to  baseband,  processed  (for  signal 
regeneration),  and  retranslated  to  RF  for  transmission.  This  is  termed  a  baseband 
repeater  and  the  one  in  this  system  is  situated  two  hops  south  of  Cincinnati. 

The  radio  system  provides  1  for  N  frequency  diversity  switching  and 
sections  of  the  radio  route  are  employed  for  switching.  The  switching  sections  in 
this  system  interface  at  the  baseband  repeater.  That  is  to  say,  all  eight  radio 
hops  south  of  the  baseband  repeater  switch  as  a  unit  upon  severe  fading  or 
equipment  failure.  Likewise,  the  two  hops  north  of  the  baseband  repeater  switch 
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To  transmit  the  digital  signal,  the  baseband  between  300  kHz  and  3.6 
MHz  on  the  radio  Is  used.  Simultaneously,  a  600  FDM  voice  channel  baseband  (one 
master  group)  Is  carried  above  3.6  MHz. 

As  Is  explained  below,  the  digital  signal  format  out  of  the  highest 
level  TOM  multiplexer  is  a  6.3  Mb/s  NRZ  waveform.  Between  this  MUX  and  the  input 
to  the  radio  modulator  is  a  subsystem  called  the  hybrid  filter.  This  filter 
accomplishes  two  functions.  First,  it  combines,  on  a  frequency  division  basis, 
the  digital  signal  and  the  analog  signal  containing  the  600  voice  channels. 

Second,  it  provides  the  signal  processing  necessary  to  condition  the  NRZ  format 
for  passage  over  the  analog  radio.  This  processing  implements  a  form  of  partial 
response  coding.  Controlled  intersymbol  interference,  generated  in  the  hybrid 
filter,  is  used  to  convert  the  two-level  NRZ  signal  to  a  three-level  analog 
signal.  The  processing  is  accomplished  in  two  discrete  steps.  The  first  step  is 
done  in  the  Tx  hybrid  filter  preceding  the  transmitter  modulator  of  the  analog 
radio.  At  the  corresponding  output  of  a  radio  section,  the  signal  from  the 
demodulator  is  passed  through  a  complementary  Rx  hybrid  filter  which  completes  the 
conversion  to  the  three-level  partial  response  format. 

In  the  Western  Union  system.  Differential  Absolute  Delay  Equalization 
(DADE)  is  used  on  the  switching  sections.  Additionally,  the  switching  is  done 
with  make-before-break  switches  which  react  in  less  than  a  signalling  interval. 

The  result  is  hitless  diversity  switching. 

Western  Union  Multiplex  System 

The  Western  Union  digital  system  was  originally  planned  and  designed 
around  a  five-level  time  division  multiplex  hierarchy.  These  five  levels  of 
multiplexers  are  depicted  in  Figure  6.2-1  below. 

The  fifth  level  of  multiplexing  (20  or  40  Mb/s  output)  was  not 
implemented  on  the  Cincinnati  to  Atlanta  system  where  the  data  channel  was  placed 
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under  voice  channels  on  an  analog  radio.  The  fifth  level  multiplexers  were 
planned  to  be  used  with  digital  radios  with  either  a  20  or  40  Mb/s  capacity. 

Both  the  ultimately  planned  and  the  actually  implemented  systems 
employed  a  hybrid  synchronizing  arrangement  wherein  the  first  two  levels  of 
multiplexing  were  synchronous  system-wide;  the  top  two  (or  three)  levels  of 
multiplexing  were  operated  asynchronously  and  employed  pulse  stuffing  capabilities 
to  accommodate  variations  in  data  rate. 

The  first  level  WU  multiplexer  is  synchronous  and  accommodates  a 
variable  mix  of  input  data  rates  ranging  from  50  through  600  b/s.  The  output  is 
programmable  to  be  either  1800,  2400,  4800  or  9600  b/s.  Bit  interleaving  is  used 
to  form  the  composite  high  rate  data  stream.  At  the  input,  sampling  is  used  and 
several  different  signal  formats  are  available  to  match  various  types  of  terminal 
equipment  interfaces. 

The  second  level  multiplexer  accepts  a  mix  of  inputs  in  the  1800  to 
9600  b/s  range  of  the  first  MUX  output.  These  signals  are  combined,  again  by  bit 
interleaving,  to  form  a  56  kb/s  output  data  stream.  This  multiplexer  is 
synchronous  and  receives  its  timing  from  the  local  station  clock.  The  input  data 
formats  are  a  Western  Union  internal  standard  and  basically  are  one-half  volt 
balanced  to  ground.  The  output  of  this  multiplexer  is  V.35  compatible. 

The  third  level  multiplexers  in  the  Western  Union  system  are 
asynchronous.  From  one  to  twenty-four  synchronous  56  kb/s  input  from  the  second 
level  multiplexers  are  accepted  and  bit  interleaved  into  a  1.544  Mb/s  output  data 
stream.  Positive  pulse  stuffing  is  used  to  accommodate  variations  in  the  rate  of 
the  high  speed  data  stream.  The  signal  formats  at  the  input  to  this  MUX  are 
V.35.  At  the  output,  the  format  is  a  T-l  signal,  i.e.,  bipolar,  return-to-zero 
waveforms  without  accompanying  clock  signal. 

The  fourth  level  multiplexer  in  the  Western  Union  network  is  also 
asnychronous  and  employs  positive  pulse  stuffing.  It  accepts  up  to  four  of  the 
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1.544  Mb/s  output  signals  of  the  level  three  MUX's  and  bit  interleaves  these  to 
form  a  6.3  Mb/s  data  stream.  This  data  stream  is  formatted  as  a 
non-return-to-zero  signal.  In  the  ClncInnati-to-Atlanta  system,  the  output  of 
this  multiplexer  was  fed  into  the  hybrid  filter  and  thence  to  the  analog  radio 
input.  In  the  system  design  for  nationwide  use,  a  fifth  level  multiplexer  was 
planned  which  would  be  used  to  provide  inputs  to  digital  radio  systems. 

The  fifth  level  multiplexer  was  planned  to  be  provided  in  two  versions, 
essentially  identical,  except  that  in  one  case,  three  of  the  6.3  Mb/s  inputs  would 
be  bit  interleaved  to  provide  an  approximate  20  Mb/s  output.  In  the  second 
version,  six  inputs  would  be  combined  to  provide  a  40  Mb/s  output  data  stream.  A 
final  design  for  this  multiplexer  was  not  developed,  thus  the  input/output 
formats,  output  data  rates  stuffing  details,  etc.  were  not  completely  specified. 

It  is  notable  that  the  Western  Union  system  design,  using  asynchronous 
operation  for  the  higher  level  multiplexers,  minimized  the  use  of  high  speed 
elastic  stores,  or  buffers.  Path  length  changes  and  any  other  multi-bit  delay 
variations  were  to  be  taken  care  of  by  the  pulse-stuffing  capability. 

Western  Union  Timinq/Synchronization  System 

The  timing  and  synchronization  plan  for  the  Western  Union  digital 
network  is  described  in  this  section. 

As  was  noted  previously,  the  Western  Union  system  utilizes  synchronous 
multiplexers  up  through  the  56  kb/s  rates.  The  MUX's  higher  in  the  hierarchy  are 
operated  asynchronously.  Thus,  the  network  is  a  hybrid  in  that  it  is  not  purely 
synchronous  or  asynchronous.  Likewise,  the  timing  and  synchronization  system 
itself  is  a  hybrid  in  that  it  was  conceived  as  a  mixture  of  both  master-slave  and 
Independent  master  types  of  systems.  The  segment  of  the  planned  system  which  was 
deployed  between  Atlanta  and  Cincinnati  was  basically  operated  as  a  master-slave 
system,  but  had  the  mixed  capability. 
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The  timing  and  synchronization  subsystem  of  the  Western  Union  network 
was  built  around  a  set  of  redundant  Disciplined  Oscillators  and  a  corresponding 
Interface  Unit.  The  Disciplined  Oscillators  are  basically  phase-locked  loops 
(PLL's),  and  the  Interface  Unit  is  a  control  circuit  which  selects  the  reference 
source  to  which  the  PLL's  are  locked. 

For  master-slave  operation  of  the  network,  a  highly  stable  oscillator 
such  as  a  rubidium  standard  is  fed  into  the  Interface  Unit  at  the  master  station. 
All  other  station  clocks  in  the  network  then  receive  their  reference  from  one  or 
more  of  the  56  kb/s  synchronous  data  channels  emanating  down  from  this  master.  At 
these  slaved  stations,  the  Interface  Unit  selects  which  56  kb/s  signal  is  to  be 
used  as  the  reference. 

For  independent  master  operation,  each  node  in  the  system  where  timing 
is  derived  has  a  Loran-C  receiver  subsystem.  This  subsystem  receives  the  ground 
wave  signal  from  the  Loran-C  system  and  integrates  over  a  long  period.  A  1  MHz 
highly  stable  signal  is  then  provided  to  the  Interface  Unit  and  thence  to  the 
Disciplined  Oscillators.  All  of  the  stations  thus  deriving  timing  from  the 
Loran-C  system  would  be  synchronized. 

The  Disciplined  Oscillators  mentioned  above  are  the  heart  of  the 
Western  Union  timing  and  synchronization  system.  The  Disciplined  Oscillators  as 
stated  above  are  phase-locked  loops.  These  PLL's  were  designed  using  a 
voltage-controlled  crystal  oscillator  and  a  third-order  feedback  loop.  Three  time 
constants  are  thus  associated  with  the  loop.  The  third  time  constant  is 
approximately  200,000  seconds  providing  a  highly  stable  output  for  several  days  in 
absence  of  an  input  reference.  The  third  order  loop  serves  to  provide  the 
Disciplined  Oscillator  with  a  drift  rate  memory  such  that  the  rate  of  change  of 
the  difference  in  frequency  between  the  Internal  VCXO  and  the  reference  frequency 
Is  remembered.  When  the  input  is  removed,  the  feedback  loop  will  continue 
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correcting  for  this  difference.  Thus,  when  the  reference  is  reapplied,  assuming 
that  its  frequency  has  remained  unchanged,  the  drift  rate  of  the  VCXO  will  have 
boon  compensated  as  if  the  reference  had  not  been  lost. 

The  other  two  time  constants  built  into  the  Disciplined  Oscillator  are 
50,000  seconds  for  the  second  loop  and  (switch  selectable)  5,  25,  or  500  seconds 
for  the  first  loop. 

Figure  6.2-2  below  is  a  functional  block  diagram  of  the  timing 
subsystem  of  the  Western  Union  system. 

Discussion  of  Western  Union  System 

The  following  discussion  provides  specific  answers  to  the  set  of 
questions  given  in  Paragraph  6.1.  Because  the  Western  Union  system  was  never  made 
fully  operational,  some  of  the  questions  are  not  directly  answerable  as  noted 
below. 

A.  Fundamental  Reasons  for  Choosing  T/S  System  As  Implemented 

There  were  several  criteria  used  in  selecting  the  Western  Union  system 
as  implemented.  Cost  was  a  primary  consideration  in  choosing  a  synchronous  system 
in  that  it  was  felt  that  the  lower  level  MUX's,  of  which  there  would  be  relatively 
large  numbers,  could  be  much  simpler  in  a  synchronous  configuration.  However,  the 
higher  level  MUX's  were  chosen  to  be  asynchronous.  The  reasoning  here  was  to 
avoid  the  relatively  large  costs  of  high  speed  elastic  stores  required  for 
synchronous  operation. 

Another  reason  for  choosing  a  synchronous  mode  for  the  lower  level 


MUX's  was  to  allow  switched  circuit  operation  eventually.  This  was  the  primary 
reason  the  Western  Union  people  chose  a  synchronous  network. 
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The  master-slave  technique,  augmented  by  an  independent  master 
capability,  was  chosen  for  its  simplicity  of  design  and  simplicity  of  operation. 

At  the  time,  it  appeared  that  this  technique,  along  with  independent  clocks,  was 
well  within  the  state  of  the  art  and  achievable  in  practice,  whereas  systems  such 
as  time  reference  distribution  and  mutual  synchronization  were  considered  (by 
Western  Union)  as  unproven  and  questionable. 

Other  types  of  systems  were  considered,  most  notably  independent 
clocks.  However,  it  was  concluded  that  the  cost  of  avoiding  periodically 
overflowing  buffers  was  low  enough  to  be  well  worthwhile.  Programmed,  or 
scheduled,  error  bursts  associated  with  resetting  buffers  was  judged  unpalatable 
from  a  marketing  point  of  view. 

The  final  choice  of  system  was  made  on  the  basis  of  engineering 
judgment  about  the  practical  aspects  and  the  associated  technical  risks.  Implicit 
in  this  judgment  was  a  performance,  cost,  and  maintainability  assessment. 
Simulations  were  not  used. 

Some  brief  consideration  of  using  the  network  T/S  system  to  distribute 
precise  time  was  given.  It  was  judged,  however,  that  no  market  existed  to  justify 
the  expense.  Being  able  to  synchronize  the  system  to  another  system  was  judged 
desirable  and  was  a  factor  in  choosing  the  master-slave  technique.  It  was  felt  at 
Western  Union  that  eventually  it  would  be  desirable  to  synchronize  with  other  U.S. 
common  carriers. 

B.  Theoretical  Design  of  Chosen  T/S  System 

There  were  several  important  aspects  considered  when  specifying  the 
Western  Union  T/S  system.  First  of  all,  the  system  was  planned  Initially  to  use 
the  data  signal-below-analog  technique  (described  earlier)  on  existing  analog 
radio  systems  and/or  ground  band  modems  and/or  leased  T-l  facilities  as  the 
transmission  media  between  nodes.  Thus  the  system  design,  including  the  T/S 
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equipment,  had  to  be  compatible  with  these  transmission  methods.  The  capability 
of  the  Interface  Unit  to  accept  56  KHz,  1.544  MHz  and  1  MHz  reference  signals 
reflects  this  requirement. 

A  second  important  criteria  in  specifying  the  T/S  system  was  the 
required  mean  time  to  loss  of  bit  count  integrity  at  a  node  when  the  source  of 
timing  reference  was  lost.  It  was  felt  that  bit  count  integrity  should  be 
maintained  for  up  to  several  days,  if  possible.  This  criteria,  of  course,  leads 
to  a  stability  specification  for  the  T/S  system.  In  the  Western  Union  network,  a 
stability  of  one  part  of  109  per  day  was  the  design  goal. 

A  cost  goal  also  was  an  important  design  criteria  for  the  Western  Union 
T/S  equipment.  In  conjunction  with  the  necessary  reliability  it  was  decided  that 
the  master-slave  system  augmented  with  a  back-up  Loran-C  receiver /monitor  was  the 
most  cost-effective  way  to  provide  station  timing  throughout  the  network. 

Itemized  cost  figures  and  goals  are  not  available,  but  an  overall  goal  of  the 
Western  Union  engineering  effort  was  to  develop  the  digital  system  such  that  data 
channels  could  be  developed  (and  services  sold)  at  rates  below  the  corresponding 
analog  derived  channels.  The  station  clock  system  ended  up  costing  approximately 
$25,000  in  1972  dollars,  and  this  was  felt  to  be  compatible  with  overall  cost 
goals. 

The  following  specifications  were  detailed  (or  specifically  not  spelled 
out,  as  the  case  may  be)  for  the  development  of  the  Western  Union  T/S  equipment. 

a.  Stability  —  long/short  term. 

Design  goal  of  +1  x  10-9/24  hours 

b.  Reliability  —  no  specific  reliability  numbers  such  as  MT8F  were 
spelled  out.  However,  it  was  felt  by  the  Western  Union  engineers 
that  the  T/S  equipment  should  be  of  such  reliability  that  the 
overall  system  reliability  would  be  effectively  controlled  by  the 
radio  systems  MTBF,  and  path  availability. 
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c.  Jitter  Reduction  --  no  specific  jitter  reduction  specifications 
were  given. 

d.  Cost  Goals  --  an  approximate  capital  expenditure  goal  of  $25,000 
per  station  clock  was  given.  This  did  not  include  spares 
provisioning  or  maintenance  costs. 

e.  0/M  Aspects  --  no  MTTR  or  other  maintenance  related  goals  were 
specified. 

The  design  of  the  signal/clock  waveforms  between  the  T/S  system  and  the 
multiplexers  and  between  the  various  multiplexer  levels  was  driven  principally  by 
the  desire  to  be  compatible  with  existing  standards.  In  most  cases,  the  standards 
were  not  formally  specified  in  the  early  70' s  but  were  de  facto  standards  as 
perceived  by  the  Western  Union  engineering  staff.  This  basically  led  to  using  the 
T-l  bipolar  format  without  separate  clock  signals  wherever  possible  in  the  system. 
C.  Practical  Design/Engineerinq  Considerations 

The  only  unusual  or  unforeseen  obstacles  encountered  in  the  development 
of  the  Western  Union  system  did  not  relate  specifically  to  the  T/S  system.  It  was 
found  that  in  the  older  radio  system  used  between  Atlanta  and  Cincinnati,  the 
klystrons  used  as  transmitter  oscillators  and  as  receiver  local  oscillators  were 
microphonic.  Mechanical  vibrations  were  translated  to  electrical  noise  on  the 
baseband  in  the  lower  frequency  spectrum,  which  caused  poor  performance  of  the 
digital  system.  Additionally,  it  was  found  that  the  phase  and  amplitude  linearity 
characteristics  of  the  radio  system  were  critical,  especially  as  the  600  channel 
voice  system  above  the  digital  signal  (on  the  baseband)  was  loaded  with  traffic. 

If  the  phase/ ampl itude  characteristics  were  not  very  well  aligned,  then 
intermodulation  products  from  the  voice  signals  fell  in-band  to  the  digital  signal 
and  again  degraded  performance. 
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All  specifications  on  the  T/S  system  were  met  in  the  design  of  the 
equipment.  It  is  not  known  what  margin  was  achieved,  i.e.,  how  much  performance 
exceeded  specification. 

D.  Initial  Field  Deployment,  Testing,  O&M 

No  unusual  problems  were  encountered  in  field  deployment  other  than 
those  of  microphonic  klystrons  and  exacting  phase/linearity  requirements  as 
spelled  out  above.  The  system  as  deployed  between  Cincinnati  and  Atlanta  was 
being  operated  as  a  test-bed.  Thus,  engineers  were  used  extensively  to  get  it 
into  operation  and  no  legitimate  measure  of  how  well  O&M  craftsmen  or  other  lower 
skill-level  personnel  could  handle  the  system  was  obtained. 

At  the  time  Western  Union  deployed  the  system,  digital  test  equipment 
was  not  widely  available,  especially  as  pertaining  to  bit  error  rate  test 
equipment.  This  caused  some  problems  but  none  peculiar  to  the  T/S  aspects  of  the 
system.  Western  Union  engineers  noted  that  non-routine  test  equipment  was  not 
required  for  the  T/S  equipment.  They  also  ventured  the  opinion  that  in  today's 
environment,  bit  error  rate  equipment  would  not  present  a  problem. 

In  retrospect,  no  changes  to  the  T/S  system  design  which  would  enhance 
or  otherwise  favorably  influence  the  field  deployment  and  testing  are  known  by  the 
Western  Union  people. 

E.  System  Operations  Experience 

As  was  previously  noted,  the  Western  Union  system  did  not  go  into 
regular  commercial  service  and  so  questions  as  to  operational  experience  generally 
are  not  answerable.  Based  on  the  several  months  of  field  trials,  the  system  was 
judged  to  be  performing  as  expected.  Technical  performance,  reliability,  and 
maintainabi 1 ity  are  felt  to  have  been  as  designed  and  no  readily  obvious  change 
would  be  apparent  in  retrospect.  No  flexibility  for  growth  or  change  requirements 
have  been  identified.  When  queried  about  requirements  now  foreseen  that  would 
have  influenced  design,  the  Western  Union  engineers  responded  that  the  deployment 
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and  growth  of  AT&T's  Digital  Data  System  (DDS)  would  probably  influence  any  such 
system  they  would  design  today.  For  interoperabi 1 ity  reasons,  and  for 
cost/simplicity  considerations,  a  design  today  would  probably  take  a 
timing/synchronization  reference  signal  from  DDS  at  one  or  more  locations  and  then 
distribute  this  over  the  Western  Union  network  in  the  master-slave  fashion.  The 
Loran-C  capability  would  not  likely  be  implemented. 

F.  Other  Information  Useful  to  POD  Planning  of  DCS 

The  Western  Union  digital  systems  design/deployment  experience  did  not 
result  in  useful  information  other  than  described  above  as  regards  T/S  aspects. 

An  opinion  was  ventured  by  the  Western  Union  engineers,  however,  with  regard  to 
overall  digital  systems.  If  possible,  DOD  would  be  well  advised  to  avoid  trying  a 
hybrid  system  such  as  the  digital  MUX’s  on  an  analog  radio.  Their  feelings  are 
that  the  most  cost-effective,  least  troublesome  route  to  follow  today  would  be  to 
initiate  any  such  system  as  a  pure  digital  network,  and  not  attempt  to  incorporate 
existing  hardware. 

6.3  Canadian  Dataroute 

The  Trans-Canada  Telephone  System  (TCTS)  is  an  association  of  eight  of 
the  largest  telecommunications  companies  in  Canada.  An  organization  within  TCTS, 
the  Computer  Communication  Group  (CCG),  is  dedicated  to  data  communications 
services.  After  initial  experimental  work  in  1971  with  a  synchronous  digital 
network,  CCG  announced  the  development  of  a  much  more  expanded  synchronous  digital 
network  called  the  Dataroute.  The  system  provides  point-to-point  service  to 
fifteen  metropolitan  areas  in  Canada.  It  is  presently  in  service  carrying 
customer  traffic.  The  network  incorporates  a  timing  and  synchronization  system 
which  Is  basically  a  variation  of  the  master-slave  technique.  The  timing  and 
synchronization  system  and  the  other  major  subsystems  of  the  network  are  described 
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Dataroute  Long  Haul  Transmission 

The  Dataroute  system  was  designed  to  utilize  existing  long  haul 
facilities  of  the  member  common  carriers  of  the  Trans-Canada  Telephone  System. 

Most  of  the  long  haul  facilities  of  these  carriers  are  4  GHz  microwave  radio 
systems.  Group  band  (12  equivalent  4  kHz  voice  channels)  modems  are  the  basic 
method  employed  in  the  Dataroute  system  to  utilize  these  microwave  facilities. 
These  modems,  used  over  the  analog  system,  provide  a  56  kb/s  channel  which  is  the 
basic  trunk  for  the  system.  The  modem  used  is  a  Western  Electric  303  data  set 
modified  to  operate  at  a  56  kb/s  rate.  The  output  of  the  modem  typically 
interfaces  with  group  level  equipment  in  a  frequency  division  multiplex  carrier 
system. 

An  alternate  way  of  providing  56  kb/s  trunks  for  Dataroute  is  by  use  of 
a  Time  Slot  Access  Unit  of  T1  facilities.  This  equipment  converts  the  56  kb/s 
signal  to  a  64  kb/s  data  stream  suitable  for  insertion  directly  into  a  PCM  voice 
channel.  This  type  trunking  is  used  in  Dataroute  wherever  T1  facilities  are 
available. 

Dataroute  Multiplexers 

Dataroute  utilizes  a  basic  56  kb/s  long  haul  transmission  network  as 
noted  above.  The  channelizing  of  this  network  is  provided  by  a  two-level  time 
division  multiplex  system  described  below. 

The  highest  level  multiplexer  is  a  bit  interleaved  machine  which  is 
synchronous  on  both  the  input  and  output.  This  multiplexer  provides  channels  for 
all  circuits  of  2400  b/s  or  greater  speeds.  The  multiplexer  can  be  programmed  to 
handle  virtually  any  combination  of  input  channel  speeds  which  are  multiples  of 
200  b/s.  The  multiplexer  in  conjunction  with  the  T/S  subsystem  as  described 
below,  also  minimizes  delay  through  the  network  by  time  aligning  all  transmit  and 
receive  frames  at  a  node.  This  capability  to  align  the  transmit  and  receive 
frames  allows  channels  to  contain  submultiplexed  channels  which  need  not  be 
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demultiplexed  as  the  composite  channel  is  patched  through  a  node  on  a 
drop- and- insert  basis.  This  is  achieved  without  adding  framing  overhead  to 
multiplex  the  subchannels.  This  time  alignment  also  allows  patching  data  signals 
only  (no  clock)  between  multiplexers. 

As  stated,  the  second  level  multiplexer  is  synchronous  on  both  input 
and  output.  The  multiplexer  is  normally  programmed  to  accept  both  a  bit  clock  and 
a  frame  clock  from  an  external  source  (the  T/S  system).  This  is  the  configuration 
used  at  nodes  where  a  station  clock  is  used  --  normally  wherever  two  or  more  links 
come  together.  However,  at  terminal  sites  which  connect  to  a  single  other 
location,  the  multiplexer  can  be  programmed  to  slave  to  a  clock  and  frame  derived 
directly  from  the  incoming  line. 

The  lower  or  first  level  multiplexers  used  in  Dataroute  are  character 
interleaved  machines  which  provide  asynchronous  low  speed  channels.  The  high 
speed  sides  of  these  MUX's  are  operated  synchronously  and  feed  into  the  low  speed 
ports  of  the  synchronous  second  level  MUX's  described  above. 

The  first  level  multiplexers  are  designed  for  a  high  degree  of 
flexibility  in  that  each  can  utilize  up  to  eight  high  speed  output  lines 
simultaneously  and  can  be  programmed  (via  PROM's)  such  that  any  of  the  (up  to)  127 
input  channels  can  be  assigned  to  any  of  the  output  ports. 

For  both  levels  of  multiplexing,  an  in-band  signalling  scheme  is  used 
to  provide  network  routing,  control,  and  testing.  Controls  that  can  be  sent  are 
the  usual  Request  to  Send,  Carrier  On,  Data  Terminal  Ready,  Data  Set  Ready,  Clear 
to  Send,  Ring  Indicator,  and  Busy  Indicator.  Test  information  such  as  loopback 
commands,  query  requests  and  responses,  and  alarm  information  are  also  sent  via 
the  in-band  signalling  system. 

In  the  next  section,  the  Timing  and  Synchronization  subsystem  of  the 
Canadian  Dataroute  is  discussed. 
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Dataroute  Timing  and  Synchronization 

From  the  beginning  of  the  TCTS  data  communications  network,  the  T/S 
system  has  evolved  through  three  stages.  Originally,  a  very  simple  master-slave 
system  was  used  with  a  rubidium  master  located  in  Toronto.  The  next  stage,  and 
that  which  is  primarily  in  operation  now  is  a  master-slave  augmented  with 
capabilities  not  found  in  typical  master-slave.  This  system  is  described  in  more 
detail  below.  This  T/S  system  has  been  in  operation  since  early  1974  in 
conjunction  with  the  subsystems  previously  described  and  today  forms  the  heart  of 
the  Dataroute.  Dataroute  people  refer  to  the  augmented  master-slave  system  as 
Hierarchical  Master  Slave  (HMS). 

Plans  are  presently  underway  in  the  CCG  to  add  a  third  level  of 
multiplex  to  the  Dataroute  and  with  it  to  overlay  a  higher  level  T/S  subsystem. 
This  "new"  T/S  network  will  again  be  essentially  a  pure  master-slave  type  which 
will  obtain  its  reference  from  the  same  basic  source  as  the  HMS  system.  The  new 
third  level  multiplexer  and  the  new  T/S  system  will  operate  at  1.544  Mb/s  and  will 
provide  channels  for  the  existing  56  kb/s  Dataroute.  The  two  timing  systems  (new 
MS  and  old  HMS)  will  organize  and  run  independently  except  for  the  common  master 
reference. 

The  new  higher  level  MUX  and  T/S  equipments  are  presently  in  the  early 
field  trial  and  deployment  stages  and  are  not  discussed  further  in  this  report. 

The  HMS  system  is  discussed  in  what  follows  and  is  referred  to  simply 
as  the  Dataroute  T/S  system. 

The  Dataroute  T/S  system  is  implemented  by  a  master-slave  system  in 
which  double  endedness  and  self  organization  are  incorporated.  (Directed  control 
is  of  course  used  as  it  is  inherent  in  master-slave.) 

The  HMS  system  is  like  a  conventional  master-slave  system  in  that 
network  timing  emanates  from  a  stable  master  source  located  somewhere  near  the 
geographical  center  of  the  network  (Toronto  in  this  case).  This  timing  is  fed 
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down  the  network  in  tree- like  fashion.  Self-organization  Is  accomplished  In  a 
hierarchical  fashion  hence  the  HMS  name.  Accompanying  the  timing  signal  is  a 
ranking,  or  figure  of  merit  signature  whose  value  at  any  location  is  generally 
dependent  on  how  remote  the  location  is  from  the  master  source.  This  signature  is 
carried  on  a  400  b/s  overhead  channel.  The  signature  is  carried  as  a  three  digit 
number  where  digit  one  designates  the  node  from  which  the  clock  first  originates. 
Digit  two  signifies  the  number  of  links  that  have  been  traversed  from  the  node 
having  the  original  (master)  clock,  and  the  third  digit  carries  the  value  of  the 
immediately  prior  node.  As  the  timing  reaches  each  node  and  is  passed  on  the  last 
two  digits  are  updated.  At  every  node  in  the  network  each  incoming  56  kb/s  stream 
carries  such  a  signature.  At  a  node  with  several  incoming  channels,  the  stream 
with  the  lowest  valued  signature  is  chosen  as  the  one  from  which  to  derive  timing 
reference.  [The  designated  master  of  the  network  is  denoted  as  node  zero  and  every 
other  node  in  the  network  is  assigned  a  numerical  value  corresponding  to  its  rank 
in  the  hierarchy.  Lower  numerical  values  denote  higher  rank.)  By  continually 
monitoring  the  incoming  signatures,  each  node  uses  that  with  the  lowest  numerical 
value  as  its  reference  source.  Thus  it  can  be  seen  that  the  network  will 
automatically  reconfigure  in  event  of  loss  of  master  or  a  link  failure;  timing 
loops  will  not  be  set  up  in  the  reconfiguration.  This  procedure  also  gives  a 
capability  to  automatically  reconfigure  upon  initialization  of  the  system  assuming 
proper  node  designations  have  been  made.  Automatic  reconfiguration  without 
setting  up  unstable  timing  loops  was  the  primary  reason  for  using  the  HMS  system 
as  opposed  to  a  straight  MS  system. 

Another  system  feature  incorporated  in  Dataroute  is  called  Master  Frame 
(MF).  This  concept  is  basically  the  dissemination  of  a  framing  epoch  marker  by 
the  timing  supply  at  each  node  in  addition  to  a  bit  epoch  marker.  In  the  56  kb/s 
system,  the  basic  frame  is  280  bits  long,  i.e.,  the  frame  epoch  occurs  at  a  200  Hz 
rate.  Thus  the  nodal  timing  supply,  or  station  clock,  delivers  to  each 
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multiplexer  both  a  bit  clock  at  56  KHz  and  a  frame  clock  at  200  Hz.  This 
delivered  frame  clock  is  the  Master  Frame.  This  common  frame  marker  is  obviously 
time  aligned  at  all  multiplexers  in  a  given  station  and  is  useful  in  minimizing 
delay  through  the  network  for  a  channel  patched  through  a  station.  This 
minimization  comes  about  because  the  low  speed  input  buffer  requirements  are 
minimized  on  each  MUX.  Also,  as  previously  noted,  the  Master  Frame  concept  allows 
drop  and  insert  of  submultiplexed  channels  without  requiring  overhead  framing  bits 
for  the  submultiplexing. 

In  conjunction  with  the  Master  Frame  concept,  another  feature  called 
Universal  Time  Frame  is  implemented  in  the  Dataroute  T/S  system.  Universal  Time 
Frame  (UTF)  is  a  further  embodiment  of  double  endedness.  This  embodiment  actually 
involves  an  interfunctioning  of  the  T/S  subsystem  with  the  second  level 
multiplexers  (56  kb/s).  The  designers  of  Dataroute  included  UTF  to  further 
minimize  network  user  delays.  They  recognized  that  this  feature  would  also  allow 
precise  time  dissemination  210  j  however,  this  capability  is  not  exploited  in 
the  system. 

Universal  Time  Frame,  as  noted  previously,  involves  both  the  second 
level  MUX's  and  the  T/S  subsystem.  Basically  the  system  operates  as  follows. 
Consider  two  communicating  nodes  in  the  network  and  the  case  where  one  of  the 
nodes  is  slaved  to  the  other  for  timing  dissemination  purposes.  The  two 
communicating  MUX's,  when  initially  put  into  service,  will  be  programmed  such  that 
each  transmitted  frame  is  advanced  in  time  (relative  to  local  master  frame)  by  an 
amount  which  slightly  exceeds  the  propagation  delay.  This  delay  is  known  from 
theoretical  or  empirical  (or  both)  considerations.  Because  the  frames  are 
advanced  at  the  transmitter,  they  arrive  at  the  far  end  of  the  link  (nearly)  in 
phase  with  master  frame.  The  receiving  multiplexer  at  the  master  station  of  the 
communicating  pair  measures  the  time  alignment  of  the  received  frame  by  measuring 
the  operating  position  of  its  high  speed  input  buffer.  This  buffer  position  is 
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then  communicated  to  the  slaved  node  via  the  overhead  channel  previously  mentioned 
(with  regard  to  the  HMS  signature  information).  At  the  slaved  node,  this 
information  is  used  to  generate  an  error  signal  to  the  local  station  clock.  This 
signal  causes  the  clock  to  adjust  its  frequency  so  as  to  advance  or  retard  the 
time  alignment  of  the  frame  marker  and  thereby  drive  the  buffer  at  the  master  node 
to  its  center  (null)  position.  Correcting  the  slaved  clock  obviously  adjusts  its 
master  frame  epoch  relative  to  the  master  station.  This  continuing,  dynamic 
correction  of  the  slaved  station's  master  frame  obviously  serves  to  keep  the 
slaved  frame  marker  time  aligned  with  the  master  station  marker.  This  alignment 
then  passes  down  through  the  network. 

The  transmittal  of  MUX  buffer  position  information  to  the  other  node 
constitutes  double  endedness. 

Discussion  of  Dataroute  System 

The  following  discussion  provides  specific  answers  to  the  set  of 
questions  given  in  Paragraph  6.1. 

A.  Fundamental  Reasons  for  Choosing  T/S  System  As  Implemented 

Efficiency  (ratio  of  data  bits  to  data-plus-overhead  bits),  ease  of 
submultiplexing  many  different  rates  of  user  channels,  and  system  simplicity  were 
the  chief  reasons  for  choosing  a  synchronous  digital  network.  The  criteria  for 
choice  of  which  type  synchronizing  system  were  largely  based  on  the  Canadian 
network  topology.  It  was  envisioned  that  a  long  thin  network  would  initially  be 
deployed  but  that  a  highly  interconnected  topology  would  evolve.  Flexibility  to 
accommodate  relatively  unpredictable  growth  was  also  considered  essential. 

As  described  earlier,  the  Dataroute  T/S  system  is  master-slave 
augmented  by  double  endedness  and  self-organization.  A  straight  master-slave 
system  as  well  as  a  mutual  synchronization  system  were  also  considered.  The 
choice  was  made  on  grounds  of  enhanced  security,  ease  of  network  expansion,  and 
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relative  insensitivity  to  link  failures.  The  choice  was  also  tempered  by  a  desire 
to  retain  simiplicity  to  the  extent  possible.  Thus  performance  was  directly 
factored  into  the  choice;  costs  and  0/M  influences  were  indirectly  factored  in. 

Referencing  the  system  directly  to  UTC  or  to  a  national  standard  was 
not  considered  as  an  explicit  need  or  requirement  for  Dataroute.  As  was  noted 
earlier,  it  was  recognized  that  the  Universal  Time  Frame  feature  allows 
distribution  of  (relatively)  precise  time,  but  the  CCG  did  not  foresee  a  need  or 
market  for  this  service  to  the  extent  that  the  additional  hardware  would  have  been 
cost  justified.  On  the  other  hand,  compatibility  with  other  systems  was  felt  to 
be  a  future  need  worthy  of  factoring  into  the  choice  of  T/S  system.  It  was 
expected  that  interfacing  with  other  synchronous  digital  networks  would  be  needed 
and  it  was  recognized  that  a  master-slave  type  system  allows  this  simply  by  using 
a  common  reference  as  the  primary  master  source. 

B.  Theoretical  Design  of  Chosen  T/S  System 

After  the  choice  was  made  as  to  the  type  of  T/S  system,  a  number  of 
factors  constrained  the  design  of  the  hardware.  The  basic  trunking  medium,  as 
described,  was  primarily  to  be  over  existing  4  GHz  microwave  systems  and  the 
design  had  to  account  for  the  characteristics  of  this  transmission  channel.  The 
rate  of  delay  variations  due  to  the  transmission  media,  the  induced  jitter  on  the 
timing  signals,  the  probabilty  of  outages  and  link  failures,  and  the 
probability/frequency  of  re-routing  of  transmission  channels  were  among  the 
factors  considered. 

Other  factors  which  led  to  specifications  on  the  T/S  hardware 
included:  the  desired  time  to  loss  of  bit  count  integrity  in  case  of  failures  in 
the  timing  dissemination  chain,  the  reliability  budget  for  the  overall  system,  and 
cost  goals  throughout  the  system. 
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Precise  values  of  hardware  specifications  are  not  available;  most  of 
these  are  considered  proprietary  of  CCG  or  else  by  the  equipment  vendors  for 
Dataroute. 

C.  Practical  Design/Engineering  Considerations 

No  significant  obstacles  were  encountered  in  the  design/development  of 
the  Dataroute  T/S  system  hardware.  Each  node  in  the  system  contains  two  station 
clocks.  Basically  these  clocks  are  made  up  of  a  highly  stable  VCXO  configured  in 
a  phase-locked  loop.  The  loop  is  designed  with  a  very  narrow  bandwidth  (nominally 
specified  in  microhertz)  and  with  variable  slew-rate  control.  Each  clock  also 
contains  the  logic  circuitry  to  accept  and  compare  up  to  32  incoming  reference 
signals  plus  signatures  and  the  logic  circuitry  to  compute  its  own  outgoing 
signature.  Additionally  of  course,  there  is  digital  frequency  synthesizer 
circuitry  for  deriving  the  various  clock  and  frame  rate  signals. 

None  of  the  above  circuitry  presented  inordinarily  difficult  design 
problems.  The  logic  speeds  involved  were  not  extreme  and  the  usual  precautions  in 
board  layout,  wiring  procedures,  etc.,  were  sufficient  to  ensure  good  performance. 

In  general,  all  specifications  on  the  hardware  were  met  or  exceeded  by 
the  final  design. 

D.  Initial  Field  Deployment.  Testing,  O&M 

Specific,  identifiable  installation  problems  were  not  encountered  in 
the  early  field  installation  of  Dataroute.  As  is  described  later,  some 
difficulties  were  met  that  can  generally  be  traced  partially  to  the  complexity  of 
the  double  ended  T/S  system  concept  and  to  the  specific  method  of  implementing  the 
concept.  These  are  discussed  more  fully  below  under  Systems  Operations 
Experience.  They  do  not  necessarily  relate  to  initial  deployment,  testing,  and 
O&M. 
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Because  the  deployment  of  Dataroute  was  planned  to  be  carried  out  as  a 
field  trial,  engineering-level  personnel  from  both  CCG  and  hardware  contractor 
organizations  were  used  extensively.  The  fact  that  the  field  craftsmen  from  the 
constituent  TCTS  organizations  were  mostly  experienced  in  FDM-FM  analog  technology 
also  influenced  the  decision  to  rely  heavily  on  engineers  for  initial  deployment. 

No  specialized  or  otherwise  non-routine  test  equipments  were  required 
in  the  early  field  deployment  of  Dataroute.  As  with  most  digital  systems,  the 
indispensable  tool  utilized  was  an  oscilloscope  that  could  be  externally 
synchronized.  A  logic  analyzer  type  of  instrument,  designed  to  interface  with  the 
multiplexers,  was  used  and  is  of  course  a  specialized  item.  This  however,  was  not 
necessarily  required,  i.e.,  the  system  could  have  been  put  into  service  without 
its  use.  More  importantly  to  this  report,  the  use  of  this  instrument  was  not 
influenced  by  the  type  of  T/S  system  in  use.  The  utility  would  have  been  the  same 
with  independent  clocks,  mutual  synchronization,  or  whatever  as  the  T/S  method. 

E.  Systems  Operation  Experience 

The  Dataroute  T/S  system  has  operated  as  expected.  Basic  specifications 
have  been  met  and  no  problems  directly  related  to  concepts  or  theory  have  been 
encountered.  However,  two  issues  have  arisen.  One  of  these  relates  to  a 
practical  problem  on  clock  distribution  in  the  terminal.  The  other  relates  more 
to  system  design  philosophy.  Both  of  these  issues  are  discussed  below. 

The  clock  (and  frame)  signals  in  a  Dataroute  node  are  distributed  from 
the  station  clock  equipment  rack  to  each  rack  of  multiplexing  equipment  by  means 
of  a  redundant  bussing  scheme.  Two  busses,  one  from  each  of  the  (redundant) 
station  clocks,  distribute  signals  to  each  rack  of  multiplex  equipment.  This 
distribution  is  accomplished  via  distribution  modules  mounted  on  the  top  of  each 
bay  (rack)  of  multiplex  equipment.  Several  problems  have  arisen  with  this. 

First,  it  was  found  that  having  the  modules  (active  devices)  mounted  above  the 
racks  presented  serious  access  problems  and  that  maintenance  and/or  expansion 
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activities  often  necessitated  interrupting  customer  traffic.  Second,  it  was  found 
that  although  up  to  six  MUX's  could  be  driven  in  tandem,  this  was  insufficient  in 
some  nodes  and  further  expansion  was  not  easily  implemented.  The  lesson  in  this 
Is  to  plan  that  the  clock  distribution  system  be  easily  expanded  and  that  ideally, 
the  expansion  capability  be  essentially  unlimited  in  size. 

Initally  also,  the  clock  signal  was  a  current  source-to-ground  format. 
This  contributed  to  the  limitations  on  the  number  of  MUX's  to  be  driven.  It  also 
gave  rise  to  interference  problems  through  noise  pickup,  and  common  mode 
coupling.  This  problem  was  eliminated  by  redesigning  with  a  format  using  a 
transformer-coupled,  bipolar,  balanced  signal  of  several  volts. 

The  second  (philosophical)  issue  relates  to  the  capability  of  the  T/S 
system.  As  has  been  described,  the  Dataroute  T/S  system  is  basically  a 
master-slave  with  three  complementing  features: 

1.  Hierarchial  Master-Slave  (self-organizing) 

2.  Master  Frame  clock  along  with  bit  clock 

3.  Universal  Time  Frame  -  a  form  of  double  endedness 

In  Oataroute,  the  reason  for  using  Hierarchial  Master  Slave  was  to 
allow  automatic  reconfiguring  of  the  network,  without  setting  up  unstable  closed 
timing  paths,  in  the  event  of  link  or  master  clock  failures.  However,  experience 
has  shown  that  network  reconfigurations  are  so  rare  that  manual  procedures  would 
doubtlessly  suffice.  In  other  words,  the  self-organizing  feature  is  so  seldom 
used  that  a  cost/benefit  analysis  based  on  this  experience  would  likely  dictate 
not  incorporating  it  in  the  design  if  the  choice  were  to  be  made  again  for 
Oataroute. 
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Note: 

The  rarity  of  failures  has  other  Impli¬ 
cations  for  military  systems.  Because 
of  the  rarity,  it  is  possible,  even  prob¬ 
able  that  personnel  will  be  unequipped 
to  respond  rapidly  and  correctly.  In 
other  words,  personnel  may  be  so  "out-of- 
practice"  that  they  cannot  diagnose  and 
correct  T/S  system  problems  in  a  timely 
manner  and  may  actually  compound  the  prob¬ 
lem  through  incorrect  activities.  This 
of  course  could  be  disastrous  in  a  wartime 
situation.  This  automatic  self-correction 
may  be  much  more  desirable. 

The  second  feature.  Master  Frame,  was  implemented  to  allow  cross 
patching  between  multiplexers  without  the  need  for  (low  speed  channel)  input 
buffers.  In  retrospect,  the  Dataroute  operators  now  question  whether  or  not  the 
ability  to  dispense  with  the  low  speed  buffers  is  a  significant  advantage.  With 
LSI  and  VLSI  usage  growing  exponentially,  the  cost  of  adding  buffering  to  channel 
cards  is  not  large.  The  simplicity  of  channel  cards  is  a  positive  feature  however 
in  that,  if  nothing  more,  it  enhances  reliability.  Additional ly,  the  ability  to 
patch  channels  with  only  signal  leads  ( i . e . ,  without  separate  clock  signals)  is  a 
positive  feature.  This  reduces  the  problems  that  arise  because  of  large  numbers 
of  channels  in  a  single  rack.  These  large  numbers  often  make  access  problems 
acute  in  that  physical  room  for  the  signal  and  clock  cables  is  scarce  and  the 
opportunity  for  wiring  errors  are  compounded. 

The  above  can  be  summarized  by  stating  that  the  use  of  Master  Frame  as 
a  way  to  avoid  the  expense  of  input  buffers  is  questionable.  The  advantage  of 
less  interbay  cabling  and  simpler,  more  reliable  channel  cards  is  positive  however 
and  perhaps  cost  justify  Master  Frame. 

The  third  significant  feature  in  the  Dataroute  T/S  system  is  Universal 
Time  Frame  (UTF).  The  original  reason  for  UTF  was  to  minimize  user  delay  through 
the  network:  UTF  does  this  by  assuring  that  the  high  speed  frame  alignment 
buffers  in  the  synchronous  multiplexers  remain,  on  average  and  in  absence  of 
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propogation  delay  variations,  at  their  center  or  null  positions.  However,  with 
the  hindsight  that  has  accrued  through  several  years  of  operating  experience,  it 
is  now  felt  by  the  Dataroute  operators  that  user  delay  through  the  network  is  not 
critically  important.  This  is  increasingly  true  as  data  users  begin  to  move  away 
from  protocols  that  employ  acknowledgement  type  transmissions  (such  as  Binary 
Synchronous  protocol).  Delay  considerations  are  also  less  critical  to  users  who 
continue  to  employ  acknowledgement  protocols  but  who  take  advantage  of  the  better 
channels  (lower  error  rates)  increasingly  available.  Longer  data  block  sizes  are 
feasible  (to  a  point)  on  the  better  channels  and  with  the  longer  blocks/fewer 
errors,  turnaround  time,  or  delay,  is  less  degrading  to  overall  throughput. 

In  summary,  it  is  now  felt  that  in  a  commercial  application  such  as 
Dataroute,  delay  minimization  to  the  degree  attempted  through  Universal  Time  Frame 
is  hardly  cost-effective.  Likewise,  as  previously  noted,  no  market  has  been 
identified  for  the  capability  to  geographically  disseminate  precise  time  as  is 
inherent  in  VJTF. 

The  reliability  of  the  T/S  system  on  Dataroute  has  proven  to  be 
degraded  by  the  complexity  of  the  station  clock.  The  HMS  function  plus  UTF  are 
implemented  at  the  cost  of  increased  complexity.  This  increased  complexity  serves 
to  cause  increased  hardware  failures  plus  contributes  to  maintainability  problems 
as  discussed  below. 

The  maintainability  of  the  T/S  system  on  Dataroute  has  proven  to  be 
significantly  more  of  a  problem  than  anticipated.  This  is  partially  due  to  the 
complexity  of  the  double  ended,  self-organizing  system  and  partially  due  to  the 
method  of  implementation.  The  system  has  been  in  operation  over  5  years  now  and 
maintenance  craftsmen  do  not  yet  totally  comprehend  the  synchronizing  hardware. 
Engineering  level  personnel  are  heavily  involved  in  T/S  system  maintenance  and 
operation.  Even  people  with  this  level  of  training/competence  appear  obscure  in 
their  understanding  of  the  system.  This  complexity  and  the  attendant  lack  of 
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familiarity  by  the  field  people  serves  to  reinforce  the  skeptical  attitudes  as  to 
the  cost  benefits  of  UTF  and  HMS.  Considerable  effort  has  been  expended  to  write 
and  rewrite  equipment  manuals  and  to  upgrade  training,  but  as  a  practical  matter, 
the  maintenance  people  continue  to  regard  the  T/S  equipment  with  considerable 
confusion  and  lack  of  understanding.  Much  of  the  comprehension  difficulties  can 
be  attributed  to  the  specific  method  of  implementing  the  double  endedness.  If, 
for  example,  one  Information  passed  from  one  end  of  the  link  to  the  other  were 
available  to  maintenance/operation  personnel  in  familiar  units  of  time  (seconds  or 
microseconds),  much  of  the  comprehension  difficulty  would  be  overcome,  (Note  that 
the  information  is  presented  as  a  dimmensionless  number  referred  to  as  "Diff"  in 
the  present  implementation). 

The  flexibility  of  the  Dataroute  T/S  system  is  judged  to  be 
acceptable.  No  restrictions  on  growth  or  change  to  the  network  are  imposed  by  the 
synchronizing  system.  It  is  notable  however  that  in  the  planning  for  the  1.544 
Mb/s  network,  which  is  a  growth  step  for  Dataroute,  a  straight  master-slave 
technique  for  T/S  is  planned.  Future  requirements,  as  now  foreseen,  appear  to  be 
for  a  longer,  less  Interconnected  network  -  one  wherein  closed  loops  are  less 
probable. 

When  questioned  as  to  the  other  advice  or  information  to  be  offered  to 
U.S.  Government/DOD  in  planning  for  T/S  of  the  future  DCS,  Dataroute  personnel 
advice  was  succinct:  simplicity  should  be  watchword.  To  the  extent  that 
survivability,  security,  and  other  specialized  military  considerations  allow,  the 
simpler  the  T/S  system  is,  the  better  will  be  the  performance.  Maintainability 
will  be  a  strong  function  of  simplicity  and  reliability/availability  will  be 
related  to  maintainability  more  than  any  other  system  parameter. 
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The  Datran  Digital  System 

Data  Transmission  Company  (Datran)  was  a  specialized  common  carrier 
company  formed  in  the  late  60's  to  design,  build,  and  operate  a  nationwide  network 
for  data  communications.  Hardware  was  built  beginning  in  1972  and  part  of  the 
system  put  into  operation  in  the  fall  of  1973.  The  company  began  at  that  time  to 
provide  switched  and  private  line  digital  services  to  customers  in  seven 
metropolitan  areas  on  the  digital  system  described  herein.  In  the  fall  of  1976, 
the  company  failed  financially,  declared  bankruptcy  and  ceased  operations.  The 
digital  system  was  subsequently  acquired  by  SP  Communications,  another  specialized 
common  carrier,  and  is  presently  operated  on  a  limited  basis. 

The  system  designed  and  built  by  Datran  is  described  in  the  following 
sections.  It  is  distinct  from  the  others  surveyed  in  this  report  in  at  least  one 
way.  The  total  network  was  newly  conceived,  designed,  and  built  as  a  digital 
system.  There  were  no  constraints  caused  by  having  to  utilize  existing  plant  or 
equipment;  nor  was  the  design  forced  into  compromise  in  order  to  have  a  voice 
channel  or  other  analog  transmission  capability.  Because  of  this  it  was  feasible 
to  consider  techniques,  including  the  T/S  system,  which  perhaps  were  not  feasible 
as  candidates  for  the  other  systems  surveyed. 

The  Datran  network,  as  noted,  provided  both  switched  and  private  line 
services;  it  is  composed  of  the  following  six  basic  subsystems: 

1.  Backbone  radio  system 

2.  Time-division  multiplexers 

3.  Timing  and  synchronization  system 

4.  Circuit  switching  system 

5.  Subscriber  access  system 

6.  Fault  alarm  and  control  system 


A 
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In  the  following  pages,  only  the  first  three  subsystems  are  described 
since  these  directly  relate  to  the  T/S  system  hardware  and  requirements.  The 
other  three  subsystems  are  not  discussed  except  as  ancillary  to  the  transmission 
subsystems. 

A  simplified  block  diagram  showing  the  radio,  multiplexer  and  T/S 
systems  of  the  Datran  network  is  shown  in  Figure  6.4-1. 

Datran  Radio  System 

The  radio  equipment  in  the  Datran  network  is  a  digitally  modulated, 
line-of-sight  microwave  system.  Both  the  6  GHz  and  11  GHz  U.S.  common  carrier 
frequency  bands  are  utilized  in  the  network.  In  both  cases,  30  MHz  bandwidth  is 
licensed  and  occupied  with  a  44.2  Mb/s  transmission  capability. 

The  radio  system  uses  8-level  PSK  modulation.  Fully  loaded,  it  accepts 
two,  phase-synchronous  21.504  Mb/s  data  signals  as  well  as  a  coherent  21.504  MHz 
clock  signal.  The  radio  system  also  accepts  a  coherent  307.2  kb/s  data  stream 
from  the  Fault  Alarm  and  Control  System  (FACS). 

The  radio  utilizes  differential  encoding  and  randomizing  (scrambling) 
of  the  bit  streams  prior  to  modulation  of  a  70  MHz  carrier.  The  baud  rate  of  the 
carrier  is  14.7456  MHz.  The  modulated  carrier  is  heterodyned  to  either  the  6  or 
11  GHz  band  for  transmitting. 

At  the  receivers  coherent  detection  is  employed.  After  detection,  the 
data  streams  are  descrambled  and  differentially  decoded.  The  307.2  kb/s  FACS 
channel  is  demultiplexed  and  the  two  21.504  Mb/s  data  streams  separated  and 
provided  to  the  two  associated  top-level  multiplexers. 

Space  diversity  is  used  on  all  paths  in  the  Datran  system.  The  path 
length  range  between  14  and  44  statute  miles.  A  fade  margin  in  excess  of  40  dB  is 
designed  into  all  paths.  A  radio  system  reliability  (end-to-end  for  a  100-hop 
reference  circuit)  equivalent  to  13  seconds  outage  per  path  per  year  is  the  design 
specification. 
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The  timing  signal  is  recovered  and  the  data  regenerated  at  every 
repeater  in  the  Datran  system.  Timing  recovery  Is  accomplished  in  14.7456  MHz 
phase-locked  loops.  The  loop  bandwidth  (3  dB)  is  1  kHz  in  the  ratio  system.  The 
free  running  stability  of  the  oscillators  in  the  timing  recovery  loops  is  1:10-6 
(short  term). 

The  radio  system  transmitter  at  every  station  is  keyed  by  the  clock 
signal  accompanying  the  data  signals.  Thus,  as  long  as  the  system  is  intact  (no 
failures),  every  repeater  is  synchronous.  In  the  event  of  a  link  or  equipment 
failure,  the  radio  downstream  from  the  failure  reverts  to  modulation  timing  based 
on  the  free  running  oscillator  in  the  receiver  timing  recovery  loop.  Thus 
non-synchronous  operation  pertains  in  all  radios  past  the  failed  link  down  to  the 
next  location  where  a  station  clock  (T/S  system)  is  operating.  That  clock  becomes 
the  timing  source  for  the  rest  of  the  network  downstream.  The  non-synchronous 
operation  of  the  links  immediately  downstream  from  the  failed  link  (to  the  next 
station  clock)  is  permitted  to  allow  system  monitoring  and  control  signals  to  be 
relayed. 

Datran  Multiplexer  System 

As  shown  previously  in  Figure  6.4-1,  a  three  level  mutliplexing  system 
is  used  in  the  Oatran  network.  The  first  level  multiplex  is  denoted  an  A-MUX,  the 
intermediate  level  is  denoted  a  B-MUX,  and  the  highest  level  is  a  C-MUX.  All 
three  Datran  multiplexers  are  synchronous  machines  (the  C-MUX  can  be  an  exception 
as  explained  later).  Each  utilizes  bit  interleaving  to  combine  parallel  low  rate 
channels  to  a  higher  speed  serial  bit  stream.  Every  multiplexer  features  elastic 
buffers  on  both  the  high  and  low  speed  Input  ports  for  jitter  reduction  and  to 
accommodate  propagation  delay  variations. 

The  first  level  A-MUX' s  provide  the  basic  customer-rate  data  channels. 
The  channels  are  4.8,  9.6,  or  19.2  kb/s.  There  are  two  switch  selectable  modes 
for  the  A-MUX.  In  the  A-l  mode,  the  high  speed  output  for  the  A-MUX  is  156  kb/s. 
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In  this  mode,  the  A-MUX  accepts  up  to  33  basic  4.8  kb/s  data  channels  or  various 
equivalent  combinations  of  4.8/9.6/19.2  kb/s  channels.  In  the  A-2  mode,  the  A-MUX 
has  the  capability  of  accepting  up  to  11.4  kb/s  channels  or  a  smaller  quantity  of 
combinations.  In  this  mode,  the  A-MUX  has  an  output  signal  of  56  kb/s. 

The  second  level  B-MUX  accepts  both  56  kb/s  and  168  kb/s  inputs  from 
the  A-MUX  and  likewise  has  two  switch  selectable  modes  of  operation.  In  the  B-l 
mode,  the  output  signal  is  at  2.688  Mb/s  and  up  to  46  input  channels  of  56  kb/s 
each  can  be  accommodated.  In  the  B-l  mode  up  to  15  input  of  168  kb/s  can  be 
accommodated.  In  the  B-2  mode,  the  output  signal  is  1.344  Mb/s  and  the  input 
capability  is  exactly  half  that  of  the  B-l  mode. 

The  C-MUX  has  a  high  speed  output  of  21.504  Mb/s.  On  the  low  speed 
side,  the  C-MUX  accepts  various  combinations  of  2.688  Mb/s,  1.344  Mb/s,  168  kb/s, 
and  56  kb/s.  Thus  both  A-MUX' s  and  B-MUX' s,  each  in  either  of  modes  1  or  2,  can 
input  to  a  C-MUX. 

The  A-MUX  and  B-MUX  each  provide  for  a  single  low  speed  port  called  the 
Line  Status  Channel.  This  channel  is  utilized  in  the  Datran  system  in  conjunction 
with  the  switched  service  and  provides  for  out-of-band  transmission  of  subscriber 
on-hook/of f-hook  information. 

Table  6.4-1  below  summarizes  the  channel  capacity  and  data  speeds  of 
each  of  the  three  levels  of  multiplexers. 

Among  the  three-level  multiplexers,  the  Level  C  multiplexer  has  the 
capability  of  accepting  asynchronous  data  signals  at  its  low  speed  inputs.  A 
positive-negative  stuffing  technique  is  employed  to  synchronize  the  speed  offset 
of  the  asynchronous  input  data  signals  to  as  much  as  +100  ppm  of  nominal  speed. 
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Operation 

Acceptable  Data 

Channel 

Out  of  Band 

High  Speed 

Mul tiplexer 

Mode 

Speed 

Capacity 

Line  Status 
Channel 

Output 

4.8  kb/s,  9.6  kb/s 

33  x  4.8  kb/s 

4.8  kb/s 

168  kb/s 

A-l 

19.2  kb/s 
(synchronous) 

or  equivalent 

Level-A 

A-2 

Same  as  above 

11  x  4.8  kb/s 
or  equivalent 

1.6  kb/s 

56  kb/s 

B-l 

56  kb/s  and 

168  kb/s 
(synchronous) 

46  x  56  kb/s 
or  equivalent 

56  kb/s 

2.688  Mb/s 

Level -B 

B-2 

Same  as  above 

23  x  56  kb/s 

28  kb/s 

1.344  Mb/s 

or  equivalent 

1.344  Mb/s, 

15  x  1.344  Mb/s 

2.688  Mb/s,  56  kb/s 

or  equivalent 

-- 

Level -C 

Mb/s 

-• 

and  168  kb/s 

plus 

21.504 

(synchronous  or 

12  x  56  kb/s 

asynchronous) 

or  equivalent 

An  advantage  of  positive-negative  stuffing  is  that  the  received  data 
channel  with  the  nominal  rate  will  not  incur  destuffing  jitter  that  is  inevitable 
with  positive  stuffing.  This  is  because  no  stuffing  takes  place  as  long  as  input 
data  rates  remain  synchronous. 

Both  bit  synchronization  and  frame  synchronization  must  be  established 
and  maintained  in  a  TDM  system.  For  the  Datran  network,  rather  sophisticated 
frame  markers  (patterns)  and  synchronization  procedures  are  used  at  each  level  of 
the  multiplexer.  This  is  for  rapid  detection  of  loss  of  synchronization  and  rapid 
reacquisition,  as  well  as  high  stability  against  bit  errors  In  the  received  data 
stream.  The  characteristics  of  frame  synchronization  are  summarized  in  Tables 
6.4-2  and  6.4-3. 
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Table  6.4-2.  Frame  Synchronization  Characteristics  (Acquisition  Time) 


A-l  MUX 

A-2  MUX 

B-l  MUX 

B-2  MUX 

C-MUX 

Subframe 

C-MUX 

Frame 

Number  of  bits  per 
frame  (N) 

210 

70 

1680 

840 

384 

28 

Number  of  bits  for 
a  frame  marker  (n) 

6 

3 

12 

9 

3 

7 

Frame  duration  (Tq) 

1.25  ms 

1.25  ms 

0.625  ms 

0.625  ms 

1/56  ms 

1  ms 

Mean  acquisition  time 
(designated  value) 

5.4  ms 

13.8  ms 

0.9  ms 

1.7  ms 

1.0  ms 

0.6  ms 

Mean  acquisition  time 
(specification) 

9.0  ms 

27.0  ms 

1.5  ms 

3.0  ms 

5.0  ms 

Table  6.4-3.  Frame  Synchronization  Characteristics 
(Detection  Time  and  Stability) 


A-l  MUX 

A-2  MUX 

B-l  MUX 

B-2  MUX 

C-MUX 

Subframe 

C-MUX 

Frame 

Mean  time  to  detect 
sync  loss  (designed) 

5  ms 

6  ms 

3.2  ms 

3.2  ms 

0.15  ms 

5.5  ms 

Mean  time  to  detect 
sync  loss 
(specification) 

27.5  ms 

82.5  ms 

4.2  ms 

8.4  ms 

6.0  ms 

Mean  time  between 
sync  loss  (designed) 

5  x  103 
days 

2  x  105 
days 

6  x  104 
days 

7  x  107 
days 

6  x  104 
days 

1  x  106 
days 

Mean  time  between 
sync  loss 
(specification) 

10  3 
days 

2  x  103 
days 

3  x  103 
days 

105 

days 

103 

days 

105 

days 

Synchronization  acquisition  time  depends  on  the  frame  structure,  the 


number  of  frame  marker  bits  per  frame  and  acquisition  procedure.  Table  6.4-2  also 
summarizes  the  calculated  mean  acquisition  time  for  each  multiplexer. 

Because  data  signals  transmitted  over  physical  facilities  are  subject 
to  delay  variations  and  induced  jitter,  each  multiplexer  in  the  Datran  system  was 
designed  with  elastic  buffers  on  the  input  ports  as  previously  noted.  Table  6.4-4 
summarizes  the  elastic  buffer  capacities  built  into  the  Datran  MUX's. 
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Multiplexer 

Data  Rate 

Elastic 

Store  Capacity 

1. 

Level-A  multiplexer 

Low  speed  input 

4.8,  9.6,  19.2  kb/s 

8  bits 

High  speed  input 

56  kb/s,  168  kb/s 

8  bits 

2. 

Level-B  multiplexer 

Low  speed  input 

56  kb/s,  168  kb/s 

8  bits 

High  speed  input 

1.344  Mb/s,  2.688  Mb/s 

32  bits 

3. 

Level-C  multiplexer 

Low  speed  input 

1.344  Mb/s,  2.688  Mb/s 

56  kb/s,  168  kb/s 

8  bits 

High  speed  input 

21.504  Mb/s 

32  bits 

In  addition  to  features  described  in  the  foregoing,  the  C-MUX's 
incorporate  a  feature  called  the  drop  and  insert  function  (DI).  In  certain 
applications,  most  notably  in  many  three-way  junction  station  on  the  system,  some 
channels  pass  through,  say  from  east  to  west,  while  others  will  branch.  The 
C-MUX's  with  the  DI  function  allow  nonbranching  channels  to  be  passed  from  one 
C-MUX  to  another  without  demultiplexing  and  remultiplexing.  When  the  DI  function 
is  used  to  interconnect  two  C-MUX's,  the  received  bit  stream  from  one  MUX  is,  in 
effect,  patched  across  to  the  transmit  section  of  the  other.  The  time  slots 
corresponding  to  those  channels  for  which  no  channel  cards  are  inserted  in  the  two 
respective  multiplexers  are  not  rewritten  in  the  transmitted  bit  stream.  For  the 
channels  to  be  dropped  or  branched,  channel  cards  are  inserted  and  then  these  time 
slots  are  rewritten  with  the  data  from  the  channel  cards. 

As  was  previously  noted,  the  Datran  system  provides  for  circuit 
switching  of  customer  channels.  This  is  accomplished  by  time-slot  rearrangement 
at  the  B-MUX  high  speed  level.  In  order  to  allow  this  switching  of  the  4.8  kb/s 
data  channel  at  the  2.688  Mb/s  B-MUX  rate,  the  location  of  individual  4.8  kb/s 
channels  of  all  A  multiplexers  must  be  identified  in  the  composite  2.688  Mb/s  data 
stream.  It  is  therefore  necessary  that  the  frame  phases  of  all  A  multiplexers  be 
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aligned  and  be  related  to  the  frame  structure  of  the  B  multiplexer  in  a  fixed 
phase  relationship,  prior  to  combination  in  the  multiplexer.  To  satisfy  this 
requirement,  an  optional  frame  alignment  unit  (FAU)  is  designed  for  use  in  the  B 
multiplexer.  These  frame  alignment  units  position  the  frame  phases  of  all 
incoming  signals  from  A  multiplexers  in  reference  to  the  control  signal  generated 
in  the  B  multiplexer.  The  FAU  is  an  elastic  store  with  a  write-address  counter 
and  the  read-address  counter  synchronized  to  the  frame  structure  of  the  A 
multiplexer  and  that  of  the  B  multiplexer,  respectively.  Maximum  memory  capacity 
is  equal  to  the  A  multiplexer  subframe  size. 

As  noted  previously,  both  A  and  B  multiplexers  have  two  respective 
modes  of  operation.  The  A  multiplexer  frame  structure  is  so  designed  that,  when 
three  A-2  mode  multiplexer  outputs  are  bit-interlaced  with  proper  phase 
relationship,  the  combined  output  steam  is  the  same  as  that  of  the  A-l  mode 
multiplexer.  This  is  to  simplify  the  switching  procedure.  The  same 
characteristic  also  exists  in  the  B  multiplexer. 

The  C-MUX's  incorporate  capabilities  in  addition  to  those  previously 
described.  These  capabilities  directly  relate  to  the  T/S  of  the  network,  and 
involve  an  interaction  of  the  station  clocks  and  the  C-MUX's  at  nodes  where  they 
are  co-located.  This  is  discussed  more  fully  in  the  next  section. 

Datran  Timing  and  Synchronization  System 

The  system  which  provides  timing  and  synchronization  for  the  Datran 
network  is  a  straightforward  application  of  the  master-slave  techniques  described 
in  Paragraph  2.1.3.  The  T/S  hardware  is  primari ly  made  up  of  an  equipment  called 
the  Datran  Station  Clock  (DSC)  which  supplies  bit  rate  clock  to  all  levels  of 
multiplexers.  However,  the  overall  network  timing  subsystem  is  composed  of 
functions  in  the  C-MUX’s,  the  DSC's,  the  microwave  radios  (as  a  timing 
dissemination  channel),  and  one  or  more  rubidium  standards  which  provide  the  basic 
reference  signal  (master  source). 
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The  typical  utilization  of  DSC's  was  depicted  previously  in  Figure 
6.4-1.  Note  that  In  general,  DSC's  are  used  at  three-way  junction  stations  and  at 
terminal  locations.  DSC's  are  not  used  at  two-way  repeaters.  The  DSC  basically 
is  a  highly  stable,  highly  reliable  disciplined  oscillator  with  the  ability  to 
"remember"  an  input  reference  frequency  for  relatively  long  periods  after  loss  of 
the  reference.  The  DSC  also  includes  a  frequency  synthesizer  for  deriving  various 
system  clock  rate  signals  from  the  basic  oscillator  frequency  and  line  drivers  for 
distributing  these  clock  rate  signals.  It  also  Includes  logic  hardware  for 
Interfacing  with  the  C-MUX's  on  a  systems  basis  and  for  controlling  its  own 
functioning.  The  DSC  provides  clock  signals  at  21.504  MHz,  2.688  MHz,  and  168  kHz. 

As  the  timing  source  for  a  station  where  it  is  located,  there  are  three 
modes  in  which  the  DSC  can  operate  depending  upon  the  input  (reference)  signal 
condition  and/or  manual  control.  The  modes  are: 

(a)  Master  Mode 

(b)  Slave  Mode 

(c)  Recovery  Mode 

Master  mode  operation  exists  under  one  of  three  conditions:  i)  manual 

selection;  ii)  when  the  input  reference  is  absent  or  outside  of  acceptable 

frequency  limits;  1 i i )  when  slave  mode  is  selected  or  when  a  SYNC  LOSS  signal 
input  is  activated  while  slave  mode  is  selected.  This  SYNC  LOSS  input  is  a  signal 
normally  associated  with  the  receive  section  of  a  colocated  C-MUX. 

The  DSC  will  operate  in  the  Slave  Mode  whenever  this  mode  is  manually 
selected  and  when  neither  of  the  second  two  conditions  described  above  for  Master 
Mode  selection  exists.  In  other  words.  If  Slave  Mode  is  selected  and  the  Input 
reference  signal  is  acceptable  and  the  SYNC  LOSS  input  is  not  activated,  the  DSC 
will  slave  to  the  Incoming  reference  signal. 

The  Recovery  Mode  is  a  transition  mode  that  allows  a  DSC  to  return  to 

the  Slave  Mode  after,  due  to  some  perturbance,  it  has  switched  to  Master  Mode  and 
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then  the  perturbance  Is  removed.  Recovery  mode  Is  always  traversed  when  going 
from  Master  Mode  operation  to  Slave  Mode  operation. 

In  general,  the  DSC  never  operates  routinely  in  the  Master  Mode.  Even 
at  the  location  of  the  network  master,  a  rubidium  standard  (plus  frequency 
synthesizer)  provides  the  reference  Input  to  a  DSC  operating  in  Slave  Mode. 
However,  when  the  DSC  is  required  to  operate  in  Master  Mode,  either  by  failure  of 
the  reference  or  by  manual  selection,  the  following  performance  is  achieved: 

Long  term  stability:  +4.5  x  10*8  per  6  months 

Short  term  stability:  +1  x  10-9  per  day 

Temporary  stability:  +1  x  10*8  -150  C  to  +550  C 

The  hardware  design  concepts  of  the  DSC  are  straightforward.  Basic  to 
the  equipment  is  a  triplication  of  all  components  that  could  cause  a  catastrophic 
failure  (except  power  supplies  which  are  redundant).  The  triplication  allows 
majority  voting  logic  to  precisely  determine  frequency  and  phase  failures  whereas 
with  simple  (double)  redundancy,  only  non-coincidence  checks  are  possible.  In 
other  words  with  double  redundancy,  when  two  signals  have  different  phases  (or 
frequencies),  it  is  not  possible  to  determine  which  one  is  correct  and  which  one 
is  in  error. 

The  heart  of  the  DSC  is  a  triple  set  of  modules  called  Timing 
Generators.  These  are  highly  stable  VCXO's,  each  in  a  phase-locked  loop 
configuration  where  the  loop  error  voltage  is  simultaneously  fed  to  the  VCXO  and 
also  digitized,  filtered  to  an  effective  0.04  Hz  bandwidth,  and  stored  in  memory. 
The  digitizing  is  done  to  a  12-bit  accuracy  which  translates  to  a  frequency 
accuracy  at  the  VCXO  of  a  5  x  10*H.  This  stored,  digitized  error  voltage  is 
applied  to  the  VCXO  in  instances  when  the  station  clock  transitions  from  slave 
operation  to  master.  Because  of  the  narrow  bandwidth  of  this  digitized  error 
loop,  input  jitter  with  frequency  content  down  to  much  less  than  1  Hz  is 
filtered.  By  virtue  of  this  jitter  reduction,  when  the  clock  becomes  a  local 
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master,  frequency  differences  between  it  and  the  normal  master  will  remain 
sufficiently  small  to  maintain  bit  count  Integrity  for  more  than  30  minutes  (based 
on  the  buffers  used  In  the  C-MUX's).  This  is  in  the  presence  of  up  to  5  percent 
rms  jitter  with  a  bandwidth  from  0.1  to  1000  Hz. 

The  bandwidth  of  the  (undigitized)  error  voltage  in  the  phase  locked 
loop  is  4  Hz.  This  translates  to  a  capture,  or  pull  in  range  for  the  Slave  Mode 
of  +1  x  10-7. 

When  the  DSC  traverses  the  recovery  mode  in  going  from  Master  to  Slave, 
it  has  the  capability  of  accepting  small  frequency  change  commands  from  an 
associated  C-MUX.  These  commands  are  used  to  alter  the  contents  of  the  memory 
storing  the  digitized  error  voltage.  These  changes  are  utilized  to  center  the 
high  speed  receive  buffer  of  the  C-MUX  during  recovery  operations.  The  rate  of 
change  is  sized  such  that  the  32  bit  high  speed  buffers  in  the  C-MUX's  will  be 
centered  to  within  1  bit  during  Recovery  Mode  operation.  The  Recovery  Mode  takes 
up  to  256  seconds  if  frequency  increase/decrease  commands  are  being  accepted.  If 
the  buffer  of  the  C-MUX  is  centered  to  begin  with  or  if  no  commands  are  received 
(for  example  the  C-MUX  to  DSC  cable  is  not  connected),  the  Recovery  Mode  requires 
approximately  75  seconds. 

Figure  6.4-2  below  is  a  simplified  functional  block  diagram  of  the 
Datran  Station  Clock.  The  triplicated  and  redundant  components  are  shown  as 
such.  Figure  6.4-3  is  a  simplified  block  diagram  of  the  Timing 
Generator/Phase-Locked  loop  Including  the  digital  filter  and  memory  functions. 

Discussion  of  Datran  System 

The  following  discussion  provides  specific  answers  to  the  set  of 
questions  given  in  Paragraph  6.1  as  they  pertain  to  the  Datran  system. 
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Fundamental  Reasons  for  Choosing  T/S  System  as  Implemented 
The  criteria  used  for  choosing  the  T/S  method  for  the  network  were 
principally  performance,  cost,  and  state  of  the  art  (practical  state  as  opposed  to 
theoretical)  prevailing  at  the  time  of  system  design.  Methods  other  than 
master-slave  were  considered.  Nonsynchronous  operation  using  independent  clocks 
and  various  types  of  pulse  stuffing  were  evaluated  as  possible  candidates. 
Synchronous  techniques  considered  included  designated  (or  independent)  master, 
mutual  synchronization,  and  of  course  master-slave.  Bit  stuffing  was  ruled  out  as 
being  too  expensive  to  implement  in  a  dedicated  circuit  system.  In  a  switched 
system,  the  cost  is  even  more  prohibitive.  Independent  clocks  was  discarded  as  a 
potential  candidate  mainly  because  the  prospect  of  periodically  overflowing 
buffers,  no  matter  how  great  the  period,  was  judged  to  be  a  severe  marketing 
disadvantage.  Of  the  types  of  synchronous  operation  considered,  master-slave  was 
chosen  for  cost,  performance,  flexibility  and  simplicity  reasons.  The  Datran 
network  was  envisioned,  even  far  into  the  future,  to  be  a  topologically  long  thin 
system  with  no  looped  subnetworks.  Because  the  basic  transmission  medium  at  44 
Mb/s  had  extreme  capacity  in  terms  of  customer  channels,  it  was  not  considered 
likely  that  alternate  routes  and  possible  loops  would  be  necessary  for  many 
years.  Too,  the  transmission  medium  was  to  be  designed  for  such  reliability  that 
the  need  for  failure- induced  alternate  routing  and  the  possibility  for  looped 
configurations  was  diminished.  Last,  since  transmission  was  to  be  entirely  over 
Datran  facilities,  control  of  routing  configurations  could  be  totally  controlled. 
All  these  reasons  enhanced  the  master-slave  choice,  without  self-organizing 
complications. 

Simplicity  of  hardware  and  the  attendant  advantages  in  maintainability 
and  reliability  also  weighed  in  favor  of  master-slave  as  compared  to  frequency 
averaging.  Independent  masters  appeared  to  be  a  viable  choice  as  regards 
simplicity  and  cost  (assuming  an  existing  system  such  as  Loran-C  as  the  basic 
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master  source).  However,  the  degree  of  dependency  on  a  system  whose  operational 
reliability  and  Integrity  lay  outside  the  Datran  operation  was  considered  a 
negative  aspect.  And  significantly.  In  the  case  of  Loran-C,  It  was  felt  that  the 
availability  of  ground  wave  coverage  was  lacking  for  the  total  geographical  area 
Datran  hoped  to  eventually  serve. 

Simulation  was  not  used  in  choosing  the  Datran  T/S  system.  Formal 
performance  and  cost  analyses  did  influence  the  decision;  however,  engineering 
judgements  on  practicality  and  probability  of  successful,  on-schedule  design  also 
weighed  heavily. 

System  reference  to  UTC  or  other  standards  was  not  considered  a  strong 
factor  in  evaluating  T/S  techniques.  It  was  recognized  that  master-slave  afforded 
relatively  easy  interfacing  with  one  other  synchronous  system  (by  utilizing  either 
a  common  reference  source  or  by  having  the  Datran  network  be  a  simple  extension  of 
the  other  network),  but  again  this  was  not  a  strong  factor  in  the  T/S  choice. 

B.  Theoretical  Design  of  Chosen  T/S  System 

The  Datran  system,  as  previously  noted,  was  designed  "starting  from 
scratch"  entirely.  Thus,  the  T/S  system  choices  were  not  constrained  by  existing 
plant  or  facilities.  On  the  other  hand,  it  was  decided  fairly  early  to  use  LOS 
microwave  as  the  transmission  media.  The  yearly  (or  longer)  and  daily  delay 
variations  over  multihop,  multicl imatic  microwave  chains  was  judged  to  be  not 
precisely  known.  Thus  the  T/S  system  chosen  would  have  to  be  versatile  enough  to 
handle  this  unknown. 

The  following  specifications  were  design  goals  for  the  Datran  T/S 

system. 

a.  Stability  (without  primary  reference) 

+1  x  10-9  per  day 
+4.5  x  10-8  per  6  months 
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With  a  jitterless  primary  reference  input,  the  T/S  station  clock 
was  specified  to  have  less  than  1  percent  rms  output  jitter. 

b.  Reliability 

The  station  clocks  in  the  Datran  network  were  specified  to  have  a 
mean  time  between  catastrophic  failure  of  greater  than  1,613,000 
hours. 

c.  Jitter  Reduction 

The  station  clocks  were  specified  to  operate  satisfactorily  over 
tandem  links  of  up  to  33  microwave  links.  To  accomplish  this,  a 
jitter  reduction  specification  of  at  least  6  to  1  (for  Input  jitter 
contained  in  a  0.1  to  1000  Hz  bandwidth)  was  stipulated. 

d.  Cost  Goals 

In  1974  dollars,  cost  goals  in  the  range  of  $50K  per  (fully 
configured)  station  clock  were  established. 

e.  0/M  Aspects 

Guidelines  on  modularity,  number  and  nature  of  visible  alarms  and 
indicators,  and  diagnostic  aids  were  established.  A  mean-time-to- 
repalr  (after  reaching  equipment  site)  of  less  than  30  minutes  was 
established  as  a  design  goal. 

In  the  design  of  timing  signal  Interfaces  and  formats  In  the  Datran 
network,  the  principal  criterion  was  performance  as  opposed  to  conformity  with 
existing  standards.  This  was  especially  true  for  interfaces  between  multiplexers 
and  between  multiplexers  and  the  radio.  At  customer  interfaces,  of  course, 
conformity  with  widely  used  standards  was  a  fact-of-llfe  necessity.  At  all 
Interfaces  In  the  Datran  network,  the  timing  signal  Is  carried  on  a  separate 
conductor  from  the  data  signal.  This  contrasts  with  the  possibility  of  having  the 
clock  signal  not  be  distinct  from  the  data  signal  as  Is  done  for  instance  in  the 
familiar  bipolar  format  of  TI  systems. 
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Practical  Deslqn/Enqlneerlng  Considerations 

Few  notably  difficult  or  unexpected  obstacles  were  encountered  In  the 
design  and  development  of  the  Datran  T/S  hardware.  The  design  of  majority  voting 
logic  for  controlling  the  triplicated  functions  in  the  DSC  required  close 
attention  to  signal  delays,  board  layouts,  etc.  Adherence  to  good  engineering 
practices  common  to  nanosecond  speed  logic  design  ensured  a  minimum  of  problems  in 
this  area  however. 

One  problem  was  encountered  in  a  systems  level  test  of  the  T/S 
equipments;  the  problem  related  to  the  ability  of  the  DSC  to  properly  hold  the 
system  frequency  when  a  jitter  corrupted  reference  signal  was  lost.  It  was  found 
that  the  frequency  spectrum  of  the  jitter  on  the  reference  was  of  critical 
importance  in  how  well  the  system  frequency  was  maintained.  The  overall 
specification  was,  using  32  bit  buffers,  bit  count  integrity  would  be  assured  for 
at  least  30  minutes  when  a  reference  with  10  percent  rms  jitter,  with  a  frequency 
spectrum  between  0.1  and  1000  Hz,  was  lost.  Recall  that  the  bandwidth  of  the  loop 
in  the  DSC  which  held  the  error  voltage,  or  "remembered"  the  system  frequency,  is 
0.04  Hz.  However,  it  was  found  that  if  the  jitter  spectrum  was  concentrated  near 
the  lower  end  of  the  specified  bandwidth,  the  loss  of  reference  would  sometimes 
leave  the  remembered  frequency  deflected  enough  to  cause  buffer  overflow  in  less 
than  30  minutes.  If  the  jitter  spectrum  were  spread  uniformly  over  the  0.1  to 
1000  Hz  band,  or  concentrated  near  the  high  end  however,  the  specification  was 
easily  met. 

This  problem  of  maintaining  bit  count  integrity  in  the  presence  of  low 
frequency  jitter  was  never  mainfested  In  actual  field  operation  but  only  In  lab 
tests  where  jittered  references  were  synthesized. 

With  the  exception  of  the  above  described  phenomenon,  all 
specifications  on  the  T/S  system  were  met  or  exceeded  by  the  completed  design. 

For  example,  a  jitter  reduction  ratio  of  greater  than  20  to  1  was  measured  in  the 


407  7  A 


field  for  the  DSC  whereas  the  specification  was  6  to  1.  A  mean  time  between 
maintenance  repairs  of  greater  than  22,000  hours  was  also  measured  In  the  field  on 
the  T/S  equipment.  This  compares  to  a  design  calculation  of  6700  hours.  As  for 
as  MTBF  to  catastrophic  failure,  no  such  failures  had  been  encountered  at  last 
check  so  that  a  meaningful  quantitative  statement  cannot  be  made. 

D.  Initial  Field  Deployment,  Testing,  O&M 

In  the  Initial  deployment  and  operation  of  the  Datran  network,  all 
field  personnel  were  given  comprehensive,  but  extremely  time-compressed  training 
on  all  equipments.  By  and  large,  the  personnel  were  experienced  in  FDM-FM  analog 
systems  as  opposed  to  digital  systems;  thus  many  of  them  encountered  and 
progressed  through  a  definite  learning  period  on  the  entire  network.  While  the 
T/S  subsystem  presented  no  more  problems  to  the  people  during  this  learning  period 
than  did  the  other  equipments,  nevertheless  the  nonfamiliarity  was  of 
significance.  The  triple  redundancy  of  the  Station  Clock  added  to  the 
nonfamiliarity  problems  to  such  as  extent  that  it  considerably  complicated  the 
understanding  of  the  equipment.  It  also  greatly  added  to  the  difficulty  of 
troubleshooting  equipment  malfunctions. 

Datran  procured  several  items  of  test  equipment  related  to  the  T/S 
system  that  were  nonroutine.  Among  these  were  portable  rubidium  standards, 
frequency  synthesizers,  and  phase  jitter  meters.  In  retrospect,  it  is  apparent 
that  these  equipments  were  not  necessary  for  operating  4nd  maintaining  the 
network.  They  were  useful  in  monitoring  the  network  performance  from  the 
point-of-view  of  an  extended  field  trial  where  continued  engineering  data  was 
desirable.  But  from  a  standpoint  of  routine,  day-in,  day-out  system  operation  and 
preventive  maintenance,  an  oscilloscope  in  conjunction  with  built-in  test 
functions  on  the  DSC  is  adequate. 

No  significant  changes  in  system  design  are  obvious  in  retrospect  that 
would  enhance  or  otherwise  favorably  influence  field  deployment  and  testing. 
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E.  Systems  Operating  Experience 

The  T/S  subsystem  In  the  Datran  network  has  operated  substantially  as 
expected.  No  significant  problems  have  been  encountered  that  would  give  reason  to 
reevaluate  the  basic  concepts  or  upgrade  the  system.  However,  it  is  probable  that 
a  simpler,  less  complex  system  would  be  designed  if  the  task  were  to  be  done 
again.  Some  reasons  for  this  are  noted  below. 

The  fully  triplicated,  highly  reliable  (and  relatively  expensive)  DSC 
was  deployed  by  Datran  at  every  three-way  junction  station  and  at  every  terminal. 
In  retrospect,  such  reliability  and  expense  is  probably  only  justified  at  remote 
three  way  stations.  Even  there,  the  need  for  triplication  is  questionable.  The 
DSC's  provide  quite  long  mean-time  between  maintenance  repairs;  and  the  nature  of 
failures  encountered  to  date  indicate  that  double  redundancy  (and  the  simplified 
control  hardware  inherent  in  this)  would  have  sufficed. 

Likewise,  experience  gained  in  actual  operations  of  the  system  tended 
to  indicate  that  early  concerns  over  jitter  on  the  recovered  timing  signals  was 
not  warranted.  In  the  actually  deployed  Datran  network,  the  longest  span  between 
DSC’s  was  eleven  microwave  hops.  Had  the  system  been  deployed  to  the  west  coast 
as  planned,  spans  of  over  thirty  hops  would  have  been  encountered.  Nevertheless, 
experience  with  the  eleven  hop  system  Indicated  that  jitter  buildup  was  much  less 
than  expected.  The  worst  measurement  for  the  eleven  hop  system  was  jitter  of  less 
than  4  percent  rms  which  would  extrapolate  to  much  less  than  10  percent  jitter  on 
a  30  (+)  hop  span. 

The  Datran  network  actually  deployed  was  a  topologically  thin,  long 
line  network  with  no  closed  circuit  configurations.  Thus  no  opportunity  ever 
arose  to  evaluate  bit  count  integrity  type  questions  on  loss  of  parts  of  the 
timing  dissemination  cnain. 
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As  noted  above,  performance  of  the  Datran  T/S  system  was  judged 
adequate.  Likewise,  reliability  was  also  more  than  acceptable  and  perhaps  over 
designed  in  some  applications. 

The  maintainability  of  the  T/S  system  probably  should  be  judged 
acceptable  to  slightly  marginal.  The  only  questionable  area  would  be  in  the  use 
of  triplicated  functions  as  noted  above.  This  tended  to  obscure  craftsman 
understanding  of  the  equipment  and  to  render  trouble  shooting  more  difficult.  It 
was  certainly  not  a  severe  problem  particularly  for  the  better  qualified,  more 
industrious  field  people  who  would  put  forth  the  effort  to  understand  the 
hardware.  Unfortunately,  such  people  are  the  exception  in  most  organizations. 

The  flexibility  for  growth/change  of  the  Datran  Station  Clock  is  judged 
acceptable.  It  was  found  that  in  one  instance,  added  driver  ports  were  needed  to 
enable  clocking  additional  multiplex  bays  in  a  station.  Careful,  tedious  work  was 
necessary  to  add  the  capability  to  an  on-line  DSC  without  disturbing  customer 
traffic,  but  it  was  accomplished.  Should  a  redesign  ever  occur,  it  would  be 
desirable  to  provide  for  easier  (physical)  expansion  of  the  drive  capability;  this 
was  not  a  major  problem,  however. 

F.  Any  Other  Information  Useful  to  U.S.  Government  POD  in  Planning 

Defense  Communications  System 

The  same  advice  regarding  simplicity  of  equipments  as  offered  by 
Canadian  Dataroute  operators  would  be  evidenced  by  the  Datran  experience.  The 
less  complex  the  system,  the  better  will  be  the  reliability  and  maintainability. 
And,  in  the  absence  of  major  design  errors,  maintainability  will  probably  impact 
performance  more  than  any  other  single  factor. 

One  other  comment  Is  pertinent.  Compared  to  the  late  sixties/early 
seventies  when  the  Datran  network  was  designed,  there  is  a  plethora  of  data  on 
digital  systems  available  now.  Each  succeeding  conference  or  symposiun  on 
broadband  communications  provides  additional  practical  experience  documentation 
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(as  well  as  theoretical  work).  Notable  useful  examples  are  LOS  microwave  delay 
data  and  multipath  fading  degradation  experience.  Obviously,  all  these  sources 
should  be  be  used  in  planning  the  future  DCS  timing  and  synchronization  systems. 
6.5  Timing/Synchronization  Planning  for  NO.  4ESS 

As  noted  In  the  Introduction,  original  plans  were  to  include 
information  on  the  Implementation  and  operational  experiences  of  Bell  System's 
Digital  Data  System  in  this  report.  (This  system  is  a  private  line  synchronous 
network  put  into  operation  in  the  ll.S.  In  1974.)  Unfortunately,  this  information 
is  not  available;  Bell  officials  decline  to  provide  such  data  due  to  the  extensive 
pending  litigation  revolving  around  the  Digital  Data  System  (DOS).  These 
officials  were,  however,  willing  to  provide  information  on  the  current  planning 
and  engineering  efforts  related  to  synchronization  of  a  network  of  large  digital 
switches  presently  being  implemented  throughout  the  U.S.  This  network  is  referred 
to  as  the  Switched  Digital  Network  (SDN).  This  information  was  provided  by  way  of 
interview  with  several  engineers  currently  involved  in  the  engineering  and 
planning  efforts.  In  this  section,  this  information  is  described. 

Nothing  in  the  way  of  operational  or  field  deployment  experience  data 
is  presented  in  this  subsection.  The  deployment  of  the  switching  system  denoted 
No.  4  Electronic  Switching  System  (No.  4ESS),  is  just  now  beginning  to  become 
widespread  and  synchronization  of  the  geographically  dispersed  network  is  still  in 
the  planning  stages.  Thus,  only  a  description  of  previous  thinking  and  current 
engineering  plans  is  available.  The  specific  questions  described  in  Paragraph 
6.1,  and  discussed  In  preceding  sections  for  each  of  the  other  systems  surveyed, 
are  not  generally  applicable  to  this  section. 

The  No.  4ESS  is  an  electronic,  software-controlled  toll  switching 
center  designed  to  tie  local  facilities  to  the  nationwide  long  distance  network. 
The  switching  center  Is  basically  a  digital  machine  which  interfaces  both  digital 
and  analog  trunks.  At  the  inception  of  No.  4ESS  deployment,  and  at  the  present 
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time,  the  analog  trunks  predominate  in  number.  As  digital  trunks  become 
increasingly  available  however,  a  trend  that  is  growing  rapidly,  the 
interconnected  No.  4ESS  systems  take  on  more  and  more  attributes  of  large 
integrated  digital  networks.  As  this  happens,  the  advantages  of  reliably 
synchronizing  the  whole  network  are  obvious.  This  is  recognized  by  Bell  and 
considerable  effort  has  been  expended  to  plan  for  this  synchronization.  Much  of 
this  planning  is  inextricably  interwoven  with  planning  and  analysis  that  has  been 
going  on  at  Bell  Labs  since  the  early  sixties.  This  early  planning  was  the 
initiation  point  in  the  discussions  with  Bell  engineers. 

In  the  early  sixties,  various  users  of  the  Bell  Telephone  System  began 
to  implement  digital  networks.  At  first,  relatively  elaborate  word  stuffing  was 
used  to  provide  a  synchronizing  mechanism  for  the  networks.  The  cost  (at  that 
time)  of  shift  registers  caused  a  change  to  bit-by-bit  pulse  stuffing  techniques 
and  eventually  to  consideration  and  analysis  of  other  synchronizing  methods. 
Considerable  attention  was  focused  on  the  concept  of  mutual  synchronization.  Many 
investigators,  both  inside  Bell  Labs  and  elsewhere,  studied  and  published  papers 
on  the  technique.  One  of  the  advantages  ascribed  to  mutual  synchronization  was 
that  it  was  administration  free.  However,  it  was  decided  by  the  No.  4ESS  network 
planners  that  this  is  not,  in  reality,  the  case.  As  an  example,  it  was  noted  that 
up  to  20,000  DS-1  (1:544  Mbps)  trunks  can  converge  on  a  No.  4ESS.  To  choose  which 
of  these  would  provide  the  timing  reference  paths  for  a  mutually  synchronized 
system  is  a  considerable  problem.  This  is  especially  true  when  the  dynamic  nature 
of  the  topology  of  these  trunks  is  considered. 

Another  aspect  of  mutual  synchronization  noted  as  significant  was  the 
dumbell  effect.  This  situation  arises  when  two  separate,  relatively  complex, 
interconnected  areas  are  connected  by  a  single  (or  thin)  trunk. 
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This  tends  to  distort  the  survivability  of  the  mutual  sync  network  and  to  compound 
administration  problems,  particularly  when  the  exact  composition  of  the  "Thin" 
trunk  is  dynamic. 

Mutual  synchronization  was  eventually  discarded  as  a  candidate 
synchronizing  method  for  three  significant  reasons.  As  indicated,  it  was 
determined  that  its  main  attribute,  i.e.,  administration-free  operation,  was  not  a 
practical  reality.  This  was  the  primary  reason  for  rejecting  the  technique. 

Also,  evolution  in  crystal  oscillator  technology  in  the  late  60's  was  a  factor  in 
moving  away  from  mutual  synchronization.  Double  oven  crystal  oscillators  with 
millidegree  temperature  stabilities  became  economic  realities.  The  resulting 
frequency  stabilities  added  to  the  case  for  master-slave  techniques. 

Additionally,  digital  phase-locked  loop  (DPLL)  technology  progressed  to  the  point 
that  economic  practicality  was  achieved.  The  significance  in  this  was  that  DPLL's 
with  "automatic"  integral-plus-proportional  feedback  was  then  achievable.  Note 
that  integral  plus  proportional  feedback  had  long  been  used,  but  the  integral 
feedback  had  been  a  mechanical  screwdriver  adjustment  (to  oscillator  free-running 
frequency).  "Feedback"  of  course  was  a  technician  periodically  nulling  loop 
stress  voltages.  Digital  implementation  of  this  function  meant  greater  automatic 
long-term  stability  in  the  oscillators  since  drift  due  to  crystal  aging  could  be 
compensated. 

Another  influencing  factor  on  synchronization  planning  during  this  time 
frame  was  that  the  DOS  network  began  to  be  engineered.  At  first,  the  objective 
was  to  build  a  totally  slip  free  network  for  DOS.  However,  the  No.  4ESS  planners 
took  the  position  that  a  totally  slip  free  DDS  was  not  practically  achievable.  As 
an  example  of  the  several  reasons  for  this  position,  most  of  which  are  of  an 
administration/operation  origin,  it  was  pointed  out  that  there  are  some  60,000 
separate  group  channel  trunks  in  the  Bell  System.  Of  these,  an  average  of  20,000 
yearly  are  dynamic  in  the  sense  that  they  are  removed  and  subsequently  returned  to 
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service  for  one  reason  or  another.  Since  these  trunk  assets  were  to  provide  the 
transmission  channels  for  the  SON,  it  was  felt  that  such  a  dynamic  makeup  would 
prohibit  practical  operation  of  a  slip  free  network. 

For  the  SDN,  it  was  eventually  agreed  that  three  system  impairments 
would  be  foremost.  These  were  i)  bit  errors;  ii)  misframing  where  multiplexers 
lose  frame  synchronization  and  iii)  slips  where  single  eight-bit-bytes  are  lost  or 
gained.  Additionally  of  course,  timing  jitter  due  to  accumulation  on  the 
transmission  lines  and  due  to  pulse  stuffing/destuffing  is  encountered.  Of  these 
impairments,  the  most  significant  in  terms  of  timing  and  synchronization  is  slip 
and  the  byte  interval  defining  slip.  Of  course,  the  longer  the  byte  interval,  the 
less  the  frequency  accuracy  requirements  on  a  timing  source  for  given  slip  rate. 

An  upper  limit  is  placed  on  byte  length  in  the  Bell  System  because  the  byte 
buffers  give  rise  to  cross-office  delay  in  switches.  Additional  delay  is  to  be 
avoided  since  it  will  require  increased  use  of  echo  suppressors.  Echo  suppressors 
effectively  control  talker  echo,  but  unavoidably  introduce  impairments  in  the 
communication  path.  This  led  to  the  8-bit  byte  which  is  125  /js  in  a  64  kb/s 
channel . 

After  the  byte  length  was  determined,  an  acceptable  slip  rate  was 
evaluated.  A  number  of  factors  were  considered  by  Bell  in  determining  a  slip  rate 
specification.  In  voice  channels,  slips  are  trivial.  However,  for  voice  band 
data  channels,  slips  can  result  in  modems  retraining  which  results  in  outages  of 
up  to  several  seconds.  Slips  also  could  have  a  severe  impact  when  Common  Channel 
Interoffice  Signalling  is  carried  over  a  slipping  channel.  The  effect  of  slips  of 
digital  data  varies  depending  on  the  application.  It  was  noted  by  Bell  that 
commercial  applications  of  secure  voice  are  foreseen  over  data  channels  in  the 
future  and  slips  could  result  in  crytographic  sync  problems. 
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The  above  and  other  con  is derat ions  led  to  the  Bell  System  eventually 
settling  on  a  specification  of  one  slip  per  5  hours  for  an  end-to-end  reference 
circuit.  This  translates  to  a  relative  clock  frequency  difference  specification 

Q 

of  1.7  X  10  .  Because  the  possiblity  of  longer  frames  becoming  desireable  in 

the  future  however,  a  tighter  stability  specification  of  1  x  10'*°/day  was 
agreed  upon. 

At  the  present  time,  the  working  specifications  of  the  timing  and 
synchronization  subsystems  for  the  No.  4ESS  network  are: 

1.  Clock  frequency  inaccuracies  of  less  than  1  X  10"10/day. 

2.  Synchronizing  unit  at  each  switch  will  phase  lock  to  an  external 
reference.  This  reference  will  be  either  a  DS-1  signal  at  1.544 
MHz  or  the  standard  Bell  System  Reference  Frequency  (BSRF)  2.048 
MHz.  Note  that  this  latter  reference  is  presently  distributed 
throughout  the  network. 

3.  The  synchronizing  system  will  be  arranged  in  a  hierarchical 
structure  with  frequency  (not  time  or  phase)  information  emanating 
from  an  atomic  standard  at  Hillsboro,  Missouri. 

4.  The  timing  system  will  have  a  highly  structured  maintenance  plan 
administered  out  of  a  centralized  location. 

Note  that  specifications  2.  and  3.  define  a  master-slave  system  when  the  DS-1 
reference  is  disseminated  throughout  the  network.  When  the  BSRF  is  used,  an 
independent  master  system  is  being  implemented. 

It  was  noted  by  the  Bell  engineers  that  the  fourth  entry  in  the  above 
specification  list  is  extremely  important  as  contrasted  to  its  innocuous 
appearance.  It  has  historically  been  found  that  maintenance  activities  are  always 
reflected  negatively  in  reliability  performance  figures  for  communications 
networks.  Significant  percentages  of  outages  are  traceable  to  maintenance 
personnel  activities.  Since  the  timing  sources  for  digital  networks  are  so 
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central  in  the  operation,  and  outages  are  potentially  so  catastrophic,  it  is  felt 
that  the  structured  maintenance  philsosophy  is  critical.  In  general,  alarming  and 
alarm  reaction  plans  for  the  timing  and  synchronization  hardware  on  No.  4ESS  will 
have  long  time  constants.  A  loss  of  reference  alarm  will,  for  example,  be 
considered  only  a  minor  alarm  for  up  to  several  minutes.  Other  reaction  times 
numbered  in  hours  are  planned.  The  central  administrator  with  network-wide  status 
visibility,  will  be  consulted  closely  on  all  alarm  responses  and  maintenance 
activities. 

Figure  6.5-1  below  is  a  simplified  block  diagram  of  the  No.  4ESS  Timing 
and  Synchronization  Unit  (TSU).  It  is  notable  that  though  there  are  three  ports, 
a  maximum  of  two  reference  signals  can  be  accepted  by  the  TSU  (by  enabling  logic 
control)  and  one  of  these  is  the  "independent"  Bell  System  Reference  Frequency. 
With  this  simple  expedient  of  only  these  references,  network  timing  loops  are 
avoided  since  it  is  not  possible  to  set  up  a  looped  configuration  in  a  network 
without  separating  the  loop  from  the  rest  of  the  network.  This,  along  with  the 
highly  coordinated  maintenance  philosophy,  will  protect  against  network  timing 
instabilities  due  to  looped  clock  distribution  paths. 

The  phase-locked  loop  in  the  TSU  incorporates  integral-plus- 
proportional  feedback.  The  integrated  component  of  the  loop  feedback  has  a  time 
constant  of  approximately  2  days.  The  time  constant  of  the  proportional  factor  is 
approximately  2  hours.  Additionally,  a  manually  activated  Fast  Start  mode  allows 
rapid  acquisition  when  the  TSU  is  first  put  into  operation  or  following  a  major 
reconfiguration  or  outage.  Automatic  phase  buildout  is  also  designed  into  the  TSU 
hardware. 

As  depicted  in  Figure  6.5-1,  processor  surveillance  and  control  Is 
implemented  for  the  TSU.  As  is  true  with  the  total  No.  4ESS  system,  extensive  use 
is  made  of  the  computer  which  controls  the  switching  machine  to  automate  technical 
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Figure  6.5-1.  Functional  Diagram  No.  4ESS  Timing  Interface  Unit 
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control  and  maintenance.  Almost  all  diagnostics,  troubleshooting,  and 
configuration  changing  is  done  via  keyboard/CRT  interface. 

The  above  briefly  describes  many  of  the  factors  which  have  entered  in 
the  planning  for  No.  4ESS  timing  and  synchronization.  The  Bell  engineers 
emphasized  that  many  of  the  choices  and  trade-offs  made  were  dominated  by  network 
administration  considerations.  This  of  course  reflects  the  culmination  of  many 
years  experience  in  operating  the  largest  common  user  communications  network  in 
the  world.  Little  or  no  consideration  has  been  given  to  some  factors  important  to 
military  strategic  communications  systems.  Survivability  in  the  presence  of 
hostile  activities  is  an  important  consideration  in  evaluating  timing  and 
synchronization  techniques  for  the  DCS.  Understandably,  this  has  had  little  or  no 
influence  on  the  Bell  designs.  Likewise,  interoperability  with  tactical  systems 
is  a  consideration  for  the  DCS  not  dealt  with  by  Bell. 

A  last  topic  discussed  with  the  Bell  engineers  during  the  interview 
concerned  the  distribution  of  precise  time.  The  T/S  system  being  engineered  for 
the  No.  4ESS  network  does  not  incorporate  double  endedness  and  thus  does  not 
readily  admit  of  precise  time  dissemination.  However,  consideration  has  been 
given  to  this  by  Bell  engineers.  In  general,  the  same  concern  for  network 
administration  considerations  influences  time  dissemination  that  influenced  T/S 
system  design.  In  fact,  the  Bell  engineers  expressed  the  opinion  that 
administration  problems  would  so  adveresly  effect  time  distribution  as  to  make  it 
impractical  in  the  Bell  System.  Two  examples  noted  were  network  topology  dynamics 
and  emergency  restoration.  The  statistic  previously  described  wherein  20,000  of 
the  60,000  group  trunks  In  the  system  are  taken  out  of  service  and  restored  each 
year  was  pointed  to  as  an  example  of  the  dynamics. 

Emergency  restoration,  and  the  tempo  of  activities  that  exist  during 
outages  was  mentioned  as  the  other  prominent  example  of  administrative  problem  in 
time  dissemination.  During  outages,  especially  those  that  effect  large 


4077A 

cross-section  trunks,  intense  effort  is  involved.  Customers  and  management  are 
usually  demanding  quick  restoral  and  little  acceptance  of  procedures  which  would 
inhibit  or  delay  restoration  would  be  anticipated.  Coordination  of  clocks,  and  of 
new  trunk  path  delays,  etc.  is  seen  by  Bell  engineers  as  just  such  a  procedure. 

It  was  noted  however,  that  automation  of  such  precedures  by  use  of  microprocessors 
might  become  economically  feasible  in  the  future. 

6.6  Sumnary  of  Survey 

The  results  of  surveys  of  three  operational  digital  networks  have  been 
described  in  the  foregoing  pages.  Additionally,  information  provided  by  AT&T  on 
plans  for  synchronizing  the  network  of  No.  4ESS  switching  centers  has  been 
reported.  In  this  present  paragraph,  all  this  information  is  summarized.  The  key 
characteristics  of  the  surveyed  systems  and  the  criteria  that  went  into  their 
design  are  presented  in  tabular  form.  This  then  is  followed  by  a  discussion  of 
how  the  design  criteria  of  these  networks  is  similar  to  criteria  for  a  strategic 
military  network  such  as  the  DCS  and  what  some  critical  differences  are. 

Each  of  the  systems  surveyed  is  a  synchronous  network,  and  each  uses  a 
T/S  technique  that  is  essentially  of  the  basic  master-slave  category  described  in 
Paragraph  2.1.3.  The  Datran  network  T/S  system  and  the  planned  No.  4ESS  network 
T/S  are  straightforward  applications  of  master-slave.  The  Western  Union  network 
and  particularly  the  Canadian  Dataroute  network  T/S  are  master-slave  with  added 
features. 

Table  6.6-1  summarizes  the  features,  specifications  and  some  commentary 
contained  in  the  main  body  of  this  section  for  each  of  the  four  systems. 

Retention  of  bit-count-integrity,  or  equivalently,  the  avoidance  of 
slips  was  the  foremost  criteria  in  the  design  of  the  networks.  Flexibility  and 
adjudged  practical  state  of  the  art  in  T/S  technology  also  weighted  heavily  in  the 
initial  system  designs  -  particularly  in  the  case  of  the  Western  Union  and  Datran 
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•A  aajor  portion  of  the  overhead  ccmbkt)  1  y  used  lor  Tioing/ Synchronization  Is  that  required  for  fra*e  synchroni  zat  ion  codes.  This  is,  however 
not  reflected  in  this  tattle. 
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networks.  Minimization  of  user  delays  over  the  network  was  a  strong  factor  In  the 
Dataroute  system  design;  recognition  of  network  acini ni  strati  on  problems  strongly 
Influenced  Bell  System  plans  and  choices. 

It  is  notable  that  only  the  Canadian  Dataroute  system  provides  for 
automatic  network  reconfiguration  (and  automatic  network  initialization)  following 
failure  or  other  perturbations.  Related  to  this  of  course  is  the  fact  that  among 
the  four  systems  surveyed,  only  the  Dataroute  requires  overhead  beyond  that 
required  for  frame  synchronization  codes. 

Each  of  the  system  designers  recognized  the  crucial  importance  of 
reliability  in  the  T/S  hardware/subsystem.  Outages  due  to  lack  of  timing  supply 
are  potentially  catastrophic  and  could  disrupt  very  large  nunbers  of  users.  Thus, 
considerable  effort  and  expense  were  expended  to  build  high  availability  into  the 
T/S  equipment.  Western  Union  and  Canadian  Dataroute  system  architects 
accomplished  this  by  using  double  redundancy  with  automatic  switchover.  The 
Datran  system  utilized  triple  redundancy  in  the  station  clock  with  all  three 
portions  contributing  in  the  unfailed,  or  normal,  operating  mode.  Majority  voting 
logic  in  effect  switches  out  the  failed  circuitry  when  problems  arise. 

The  field  deployment  experiences  of  the  organizations  surveyed  were 
remarkably  similar.  Each  of  the  Wetern  Union,  Dataroute,  and  Datran  intial 
deployment  efforts  were  reasonably  straightforward,  but  extensive  use  was  made  of 
engineering-level  personnel  (as  opposed  to  technicians  or  craftsmen).  The  lack  of 
a  ready  supply  of  digitally  oriented  field  personnel  was  noted.  In  each  case 
however,  the  T/S  system  type  or  its  features  had  little  or  no  influence  on 
deployment  difficulties  or  lack  thereof. 

In  the  area  of  operations  experience,  only  Datran  and  Dataroute  were 
able  to  contribute  information.  The  Western  Union  network  never  went  operational 
and  the  Bell  System's  deployment  of  No.  4ESS  Is  only  now  underway.  The  Dataroute 
operational  experience  has  been  adversely  effected  to  some  extent  by  the 
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conceptual  (and  hardware)  complexity  of  the  T/S  system  and  to  implementation 
procedures.  To  date,  field  personnel  are  not.  totally  comfort ahlo  with  their 
knowledge  and  understanding  of  the  method  of  timing  and  synchronizing  the  system 
with  the  extra  features  of  the  master-slave  system.  Notably,  the  next  evolution 
of  the  Dataroute  system  is  planned  to  be  an  unadorned  master  slave. 

In  the  case  of  Datran  operational  experience,  no  significant  problems 
arose  which  were  attributable  to  the  T/S  concept.  However,  the  triple  redundancy 
in  the  hardware  did  complicate  craftsman  understanding  and  troubleshooting. 

Failure  rates  were  so  low  as  to  call  into  question  the  need  for  triples  as  opposed 
to  double  redundancy. 

The  above  briefly  summarizes  the  information  gained  in  the  survey  of 
commercial  digital  networks.  Concisely  stated,  the  net  result  is  that  the  lesson 
to  be  learned  is  that  the  T/S  systems  should  be  as  simple  and  straightforward  as 
possible.  However,  it  should  be  noted  that  several  key  criteria  which  greatly 
influence  the  design  and  (satisfactory)  operation  of  a  strategic  military  network 
such  as  the  Defense  Communication  System  as  absent  from  the  list  of  commercial 
network  criteria.  Among  these  are: 

1.  Survivability  (Paragraph  6.3.1) 

2.  Compliance  with  Federal  Standard  1002  (Paragraph  2.3.6) 

3.  Intersystem  Operability  (Paragraph  2.3.9) 

4.  Self-Reorganization  (Paragraph  2.3.11) 

5.  Precise  Time  Availability  (Paragraph  2.3.12) 

In  the  above  list,  survivability  is  the  most  significant  entry.  As 
pointed  out  in  Paragraph  2.3.1,  the  fact  that  a  network  plays  a  central  role  in 
national  defense  means  that  survivability  must  be  as  pivotal  in  design  trade-offs 
as  are  any  other  parameters.  This,  of  course,  can  completely  alter  the 
conclusions  of  a  cost-effectiveness  study. 
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It  is  doubtful  that  any  of  the  network  T/S  systems  surveyed  in 
themselves  would  provide  a  survivability  measure  adequate  for  DCS  needs.  The 
Canadian  Dataroute  has  the  i nherent  capability  to  be  survivable  to  some  extent 
through  its  reorganization  feature  (hierarchical  master-slave).  However,  the 
system's  simple  free  running  stability  (of  each  node)  may  or  may  not  be  adequate 
to  provide  slip  free  operation  in  a  disconnected  node  for  the  time  period  deemed 
necessary  (24  hours). 

Precise  time  availability  is  another  feature  that  is  of  considerable 
value  in  a  defense  communications  scenario  but  of  lesser  importance  in  the 
commercial  world.  Covert  comnuni cat ions  requirements  lead  to  spread  spectrum  and 
cryptographic  system  use.  Precision  knowledge  of  a  coordinated  time  standard,  of 
course,  greatly  facilitates  the  synchronization  of  such  systems  by  reducing 
acquisition  search  windows.  Likewise,  a  knowledge  of  relatively  precise  time 
facilitates  navigation  and  position  fixing  -  another  distinct  advantage  in  a 
military  environment  but  of  lesser  importance  for  commercial  applications. 
[Although  the  commercial  need  for  precision  navigation  aids  should  not  be 
minimized.] 

The  commercial  networks  surveyed  have  made  no  attempt  to  use  their  T/S 
systems  in  any  manner  to  augment  or  form  a  basis  for  network  monitoring  and 
evaluation.  This  is  of  course  an  area  where  the  benefits  could  be  of  as  much 
importance  to  a  commercial  system  as  to  a  military  one.  Such  a  use  of  the  T/S 
system  would  generally  require  preplanning  and  engineering  of  the  capability  into 
the  hardware  during  initial  design,  something  obviously  not  included  in  the 
systems  surveyed.  Recognition  of  the  benefits  to  be  obtained  through  monitoring 
and  evaluation  could  influence  the  type  of  T/S  system  chosen.  Some  T/S  methods 
are  more  compatible  and  synergistic  with  monitoring  and  evaluation  than  others. 

A  concluding  comment  is  in  order.  Simplicity  and  administration 
related  issues  were  mentioned  over  and  over  by  the  systems  operators.  It  was 
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emphasized  that  these  aspects  should  be  central  in  the  design  of  any  T/S  system. 
The  reasoning  behind  these  statements  is  not  presently  debatable,  especially  among 
experienced  systems  operators.  However,  the  influence  that  microprocessor 
evolution  is  having  and  will  continue  to  have  on  this  aspect  should  not  be 
overlooked.  Increasingly,  it  is  feasible  to  automate  activities  heretofore 
constrained  to  be  manually  accomplished.  With  automation,  or  machine  driven 
activity,  the  rationale  behind  simplicity  arguments  should  be  re-examined. 

Without  the  factor  of  human  error,  much  in  the  way  of  automatic  corrective 
routines  are  feasible.  And  much  in  the  way  of  system  features  now  considered  very 
useful,  but  too  complex  to  be  practical,  become  realistic.  This  can  and  certainly 
should  be  factored  into  any  current  T/S  systems  design. 
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CONCLUSIONS 


The  results  of  this  study  lead  to  the  recommendations  that  the  Improved 
Time  Reference  Distribution  technique  be  employed  at  the  major  modes  in  the  future 
Switched  Digital  Defense  Communications  System.  Table  3.4  indicates  that  this 
technnique  ranked  equal  to  or  better  than  any  other  technique  in  its  ability  to 
provide  ten  of  the  twelve  desirable  characteristics.  The  two  characteristics 
where  Improved  Time  Reference  Distribution  was  not  equal  to  or  better  than  other 
techniques  in  that  table  were  in  the  requirements  for  overhead  dataspace  and  in 
the  need  for  microprocessor  execution  time  and  memory.  Section  3. 3. 7. 6  shows  that 
the  requirement  for  overhead  shown  in  colulmn  7  of  Table  3.4  is  negligible  when 
framing  synchronization  requirements  are  added  (as  it  must  be)  to  all  systems  and 
in  fact  more  precise  time  at  nodes  in  a  network  may  even  reduce  the  requirements 
for  framing  synchronization  information;  but  this  is  dependent  on  definitions  of 
requirements  for  frame  synchronization. 

In  the  case  of  microprocessor  time  and  memory  the  impact  of  increased 
requirements  for  implementing  Improved  Time  Reference  Distribution  is  reflected 
directly  in  the  cost  information  presented  in  Section  4  of  this  report.  Table 
4. 3. 2. 3-3  provides  this  information  which  indicates  that  the  estimated  cost  to 
design,  implement  and  install  and  maintain  a  200  node  network  for  20  years  will 
cost  $14,751,000  using  the  Improved  TRD  technique.  This  is  30  percent  higher  than 
the  simplest  Master  Slave  technique  or  the  Independent  Clock  technique  using 
Rubidium  clocks  which  constitute  the  two  least  expensive  systems  considered. 
However  some  additional  comments  are  in  order  relative  to  less  expensive  systems. 

Independent  clock  techniques  do  not  allow  uninterrupted  communications 
for  more  than  a  limited  period  of  time.  All  commercial  communications  designers 
have  rejected  this  approach  as  extremely  undesirable  because  of  this  specific 
limitation.  If  some  form  of  scheduling  of  interruptions  and  or  notification  of 


users  of  the  system  of  impending  interruptions  is  not  implemented  this  limitation 
could  be  costly  in  many  unconsidered  ways.  This  form  of  system  management  and  its 
instrumentation  and  control  were  not  considered  in  the  prices.  Also  while 
Rubidium  clocks  in  an  independent  clock  technique  appear  cheap  the  cost  of 
periodic  calibration  of  those  clocks  was  not  included  in  the  pricing.  Calibration 
costs  are  reduced  using  cesium,  but  the  total  life  cycle  costs  are  only  7  percent 
under  the  Improved  TRD  technique  and  periodic  interruptions  of  communications  are 
still  not  eliminated. 

The  total  life  cycle  cost  analysis  indicates  that  the  initial 
acquisition  cost  for  the  Independent  Clock  Techniques  is  greater  than  for  any  of 
the  disciplined  clock  approaches,  but  that  the  disciplined  clock  techniques 
require  greater  life  time  maintenance.  A  much  more  detailed  analysis  of  this 
costing  tradeoff  should  be  considered  (much  more  detailed  than  that  afforded  by 
the  RCA  Price  Model)  before  anyone  jumps  to  the  conclusion  that  the  Independent 
Clock  Technique  is  more  cost  effective.  Still  another  reason  why  the  Independent 
Clock  Technique  looks  cheap  is  that  it  was  assumed  no  disciplining  type  hardware, 
and  no  overhead  data  would  be  required.  This  means  that  costing  did  not  consider 
the  addition  of  hardware  and  transmission  data  space  (overhead)  which  would 
provide  for  system  monitoring  and  system  management.  As  will  be  pointed  out 
below.  Bell  System  engineers  stated  that  they  considered  that  a  Mutual  Control 
System  would  require  a  high  degree  of  monitoring  and  system  management.  It  is 
believed  that  this  statement  also  applies  to  an  Independent  Clock  Technique. 

Mutual  Control  seems  to  have  fascinated  many  who  have  seriously 
considered  the  problem  of  system  synchronization.  A  review  of  the  literature 
indicates  that  much  effort  has  gone  into  the  study  and  analysis  of  Mutual 
Control.  It  appears  that  nearly  half  the  written  reports  on  this  subject  were 
generated  by  Bell  System  Engineers;  however,  the  Bell  System  has  not  chosen  to 
implement  this  technique.  Direct  statements  by  Bell  Systems  Engineers  state  that 
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they  feel  such  a  system  would  require  extensive  system  management  with  associated 
monitoring  hardware  and  system  control  overhead  information  not  considered  in 
comparing  these  systems  in  this  study.  With  the  exception  of  one  four  node  network 
reported  on  in  Reference  196  no  one  appears  to  have  committed  this  approach  to  an 
operational  system.  A  four  node  network  could  hardly  be  considered  a  large  highly 
connected  network. 

The  area  of  most  concern  in  a  Mutual  Control  approach  is  stability  and 
interaction.  In  the  simplest  single  ended  Mutual  Control  approach  the  average 
system  frequency  is  a  weighted  average  of  the  free  running  frequencies  of  all 
nodal  clocks  in  the  network,  plus  a  term  which  is  a  function  of  the  delays  in  all 
transmission  paths.  The  weighting  of  the  free  running  frequencies  of  the  nodal 
clocks  is  a  function  of  connectivity  (topology)  and  of  the  loop  gains  in  the 
control  loops  of  each  node.  Double  endedness  eliminates  the  term  in  the  average 
system  frequency  expression  which  is  dependent  on  transmission  time  delay. 

However,  in  the  double  ended  Mutual  Control  system  the  average  system  frequency  is 
even  more  sensitive  to  topology  and  loop  gain  variation  at  highly  connected  nodes. 

All  simulations  of  dynamic  response  of  Mutual  Control  systems  in  this 
study  and  in  Reference  190  have  assumed  fixed  control  loop  gain  and  phase 
detectors  with  unlimited  range.  No  serious  study  has  gone  into  what  happens  to 
the  system  when  buffers  at  a  node  overflow.  Reference  188  shows  that  a  Mutual 
Control  system  has  a  multiplicity  of  stable  operating  states  and  in  a  large 
practical  system  it  would  be  difficult  to  predict,  let  alone  control,  the 
operating  conditions  a  system  would  settle  to  after  a  major  perturbation. 

It  has  often  been  stated  that  a  Mutual  Synchronization  system  is 
"inherently  self  organizing"  and  this  appears  to  be  true  if  a  system  uses  equal 
(or  fixed)  weighting  of  averaging  inputs  to  a  node  to  determine  the  local  clock 
control  (irrespective  of  topology  or  clock  quality.)  However,  refinements  to 
Mutual  Control  such  as  the  use  of  a  single  master  node  to  stabilize  average 
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network  frequency  or  the  application  of  Unequal  Weighting  to  allow  better  quality 
clocks  to  have  a  greater  influence  on  average  system  frequency  would  require 
system  monitoring  and  system  control  overhead  to  control  organization  when 
topology  changed. 

Thus  when  refinements  are  added  to  a  Mutual  Control  System  to  eliminate 
some  of  the  undesirable  characteristics  the  "inherent  self-organization  ability" 
disappears  and  what  advantage  remains  to  suggest  that  Frequency  Averaging  is  a 
better  choice  than  a  Master  Slave  system? 

The  Mutual  Control  System  distributes  timing  over  multiple  paths. 
However,  a  Master  Slave  System  using  Phase  Reference  Combining  serves  the  same 
purpose  and  it  can  be  designed  to  eliminate  all  feedback  paths  to  make  the  timing 
system  more  stable.  In  a  Mutual  Control  System  it  has  been  shownl90  that  a 
phase  perturbation  not  only  travels  in  all  directions  in  a  network  but  can  be 
reflected  off  the  edges  of  the  network  and  travel  back  through  it  much  as  a  ball 
bounces  off  a  wall  and  returns  toward  the  source.  In  a  master  slave  system 
perturbations  can  only  travel  outward  and  cannot  bounce  back. 

Reference  to  the  frequency  plots  of  the  performance  of  all  Mutual 
Control  Systems  in  this  study  show  that  the  avet age  system  frequencies  do  not  vary 
over  the  simulation  runs.  But  it  should  be  emphasized  that  in  all  these 
simulations  these  plots  are  for  frequency  and  phase  variations  WITH  RESPECT  TO 
NODE  1.  Since  this  is  true,  the  center  line  (absissa)  of  the  graphs  do  not  show 
how  much  Node  1  is  changing  as  a  function  of  time  relative  to  a  precise  time 
reference. 

When  Node  1  is  assumed  to  be  an  uncontrolled  high  qual ity  master  clock 
and  the  average  system  frequency  of  the  remaining  Mutual  Control  nodes  is  drifting 
relative  to  the  master  it  is  shown  that  phase  differences  in  the  network  will 
accimulate  without  bound  until  buffer  overflow  and  communications  are  interrupted. 


7809A 


7-4 


i 


L 


$ 


In  all  the  literature  covering  the  subject  of  Mutual  Control  there  has 
only  been  a  few  suggested  reasons  for  choosing  it  over  Master  Slave  or  Directed 
Control  Systems.  One  of  the  earliest  reasons  was  that  it  was  believed  that 
transmission  time  delay  changes  would  be  balanced  by  "equal  changes"  in  elastic 
storage  through  the  network.  Further  analysisl88  disproved  this.  Automatic 
self  organization  was  another  major  reason  for  considering  its  use.  However, 
other  refinements  to  Mutual  Sync  considered  desirable  from  a  performance 
standpoint  make  the  system  no  longer  have  an  inherent  self-organizing  capability. 

The  requirement  that  every  active  link  into  a  node  must  be  instrumented 
such  as  to  contribute  to  the  control  of  the  local  node  makes  a  Mutual  Control 
System  more  complex  and  expensive  than  a  Directed  Control  System  which  contains  a 
similar  degree  of  refinement  as  shown  in  the  pricing  comparisons  at  the  end  of 
Section  4  of  this  report. 

The  Harris  Program  Manager  on  this  study  has  spent  over  a  dozen  years 
involved  in  considerations  of  methods  of  network  synchronization  starting  as 
principal  investigator  of  the  Mutual  Control  approach  under  a  Mallard  System 
Synchronization  Studyl93  in  1968.  In  that  time  there  has  never  been  a 
defendable  reason  suggested  for  the  use  of  Mutual  Synchronization.  Not  in  the 
light  of  the  fact  that  a  Directed  Control  System  can  have  features  added  which  can 
make  it  self  organizing  and  able  to  obtain  timing  information  over  more  than  one 
path  simultaneously. 

The  Harris  staff  which  pursued  this  study  has  come  to  the  conclusion 
that  Mutual  Control  Synchronization  offers  NO  advantage  over  a  Directed  Control 
(Master  Slave)  System  if  at  least  some  of  the  refinements  of  the  Improved  Time 
Reference  Distribution  System  can  be  included.  If  further  studies  are  to  be 
conducted  relative  to  the  trade-offs  between  Mutual  Control  versus  Directed 
Control  it  is  strongly  suggested  that  the  topology  and  size  of  the  system  in  a 
normal  operating  mode  be  specified  along  with  the  topology  and  size  of  the 
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network  after  failures  or  destruction  of  portions  of  the  network.  This  is  because 
the  performance  of  a  Mutual  Control  System  is  so  much  dependent  upon  topology  and 
number  of  nodes.  Anyone  contemplating  the  use  of  Mutual  Control  should  be  warned 
not  to  jump  to  conclusions  about  relative  performance  between  a  Mutual  Control 
System  and  a  Directed  Control  System  based  on  experimental  results  obtained  from  a 
network  of  only  a  few  nodes.  A  Mutual  Control  network  of  many  nodes  and  worldwide 
geographical  distances  may  react  very  differently  to  one  of  a  few  nodes  and 
shorter  distances.  Remember  that  transmission  time  delay  and  its  variations  are 
parameters  appearing  in  system  stability  and  in  accumulated  phase  difference 
(buffer  size)  equations. 

The  recommendations  boil  down  to  the  use  of  Directed  Control  (Master 
Slave)  synchronization  using  those  features  which  have  been  defined  to  constitute 
an  Improved  Time  Reference  Distribution  synchronization  system.  In  this  study  we 
have  attempted  to  evaluate  the  value  of  each  of  those  features.  Certainly,  as 
proven  in  the  Datran  and  Western  Union  systems,  a  simple  Master  Slave  approach 
will  work.  However,  what  is  to  be  gained  when  Double  Endedness  is  applied.  Look 
at  all  the  graphs  in  Appendix  C  and  study  the  maximum  phase  errors  (variations)  of 
both  Directed  Control  and  Mutual  Control  Systems  which  do  not  use  Double 
Endedness.  In  a  system  in  which  satellite  links  are  involved  these  variations 
range  from  20  to  30  microseconds.  Now  study  the  same  phase  plots  for  the  same 
systems  after  Double  Endedness  is  incorporated.  The  maximum  phase  variations  are 
reduced  to  some  2  to  3  microseconds.  Thus  Double  Endedness  provides  for  about  a 
10  to  1  reduction  of  the  phase  differences  at  various  nodes  in  the  network. 

Datran,  Western  Union  and  the  Canadian  Dataroute  did,  or  do,  not  use  satellite 
links,  but  the  DCS  will  use  them.  A  system  in  which  a  failure  is  not  considered 
normal  might  pass  up  the  need  for  this  phase  stability  but  the  DCS  must  continue 
to  operate  in  a  wartime  environment  with  one  or  more  nodes  attempting  to  continue 
operation  even  when  timing  information  may  be  temporarily  cutoff.  The  ability  to 
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continue  to  run  at  the  last  known  frequency  before  timing  system  failure  will  be 
dependent  on  the  quality  of  the  master  clock,  the  ability  to  eliminate 
transmission  time  delay  and  its  variations  from  effecting  the  local  clock  and  the 
quality  of  the  local  clock.  Double  Endedness  minimizes  the  effects  of 
transmission  time  delay  and  its"common  mode"  variations.  Double  Endedness 
produced  the  greatest  reduction  in  phase  variations  of  all  the  suggested  features 
considered.  If  any  feature  is  to  be  added  to  a  simple  Master  Slave  System  Double 
Endedness  produces  the  greatest  reduction  in  phase  variations  between  nodes  that 
otherwise  result  from  transmission  time  delay  variations.  This  contributes  most 
heavily  to  the  ability  of  a  node  to  free  run  in  synchronization  with  the  rest  of 
the  network  when  it  is  temporarily  cutoff  from  timing  information  input. 

Self  organization  provides  the  system  the  ability  to  automatically 
reconfigure  the  timing  distribution  system  to  allow  communications  to  continue 
between  surviving  portions  of  the  network.  Like  Double  Endedness  it  is  an 
important  contributor  to  survivability  and  is  considered  on  par  with  the 
importance  of  Double  Endedness.  Canadian  Dataroute  instrumented  a  system  of  Self- 
Organization  and  as  reported  in  Section  6  of  this  report  made  the  statement  that 
it  excessively  complicated  the  system  and  the  future  upper  level  multiplex  system 
which  they  are  now  planning  will  not  have  that  feature.  They  stated  that  the 
system  used  this  capability  so  infrequently  that  when  it  happened  operating 
personnel  did  not  understand  what  or  why  it  occurred  or  what  to  do  about  it.  The 
question  which  should  have  been  posed  next  to  these  people  should  have  been  "but 
what  if  the  system  did  not  have  the  capability,  what  would  your  system  or  your 
operating  personnel  have  done?”  If  the  self-organizing  capability  had  not  been 
there,  there  might  have  been  a  total  collapse  of  the  network  timing  system  and  the 
operating  personnel  would  have  not  only  needed  to  know  what  had  happened  but  also 
would  be  expected  to  know  the  "Optimum  method"  of  setting  up  the  system  after  the 
failure. 
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time  dissemination.  During  outages,  especially  those  that  effect  large 
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The  Bell  System  engineers  recognize  this  problem  and  state  that  they 
expect  to  make  their  4ESS  system  nodes  so  stable  that  they  can  coast  through  a 
couple  of  days  of  timing  input  failure.  When  a  failure  occurs  the  normal 
operating  personnel  at  a  node  will  be  instructed  to  do  nothing  but  make  sure  a 
team  of  operating  engineers  at  a  central  location  are  aware  of  the  failure.  This 
centrally  located  team  will  analyze  the  problem  and  instruct  personnel  deployed  at 
the  nodes  how  to  take  corrective  action. 

In  a  defense  related  system  the  central  located  engineering  analysts 
become  a  single  point  failure  situation.  The  Defense  Comnunicati  ons  System  must 
be  engineered  such  that  the  system  can  automatically  switch  to  the  best  alternate 
master  clock  through  the  best  available  path.  If  properly  engineered  and  reliably 
instrumented  the  majority  of  the  system  operating  personnel  need  not  know  why  or 
when  self-organization  happens.  System  monitoring  at  a  few  different  locations 
can  be  used  to  see  if  the  self-organization  system  is  performing  as  expected. 

This  study  indicated  that  the  feature  Independence  of  Clock  Error 
Measurement  and  Correction  did  contribute  to  improved  phase  and  frequency  accuracy 
between  the  master  and  nodes  in  the  network  as  anticipated.  However  the  degree  of 
improvement  versus  the  cost  of  incorporation  might  need  more  extensive  detailed 
study.  The  cost  to  incorporate  this  feature  is  not  great  since  all  the 
information  is  available  at  each  node.  The  hardware  implementation  is  only  what 
it  takes  to  get  this  information  into  the  data  stream  and  out  to  the  nodes  which 
need  it.  A  small  amount  of  software  will  then  allow  the  nodal  microprocessor  to 
take  this  information  into  account  when  determining  the  local  clock  error. 

On  the  other  hand,  ICEM&C  is  only  effective  during  transient  conditions 
in  the  system.  If  control  loop  time  constants  are  very  long  compared  to 
transients,  and  if  the  size  and  direction  of  transients  are  random  in  nature  then 
ICEM&C  may  not  contribute  much  when,  and  if,  a  performance  to  cost  ratio  is 
studied  relative  to  inclusion  of  this  feature.  This  deserves  more  study. 
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The  results  of  analysis  of  Phase  Reference  Combining  are  similar  to 
statements  made  relative  to  ICEM&C.  In  this  study  the  value  of  PRC  in  reducing 
measurement  jitter  was  evaluated  using  the  simulation  model.  Although  the 
conclusion  was  that  there  was  some  improvement  the  difference  was  so  small  as  to 
make  the  use  of  PRC  of  questionable  value  in  the  system  that  was  simulated.  Like 
the  ICEMtC  feature  the  cost  to  implement  this  feature  was  also  very  small  making 
it  difficult  to  say  whether  the  feature  is  cost  effective.  It  appears  that  this 
feature  should  offer  further  improved  measurement  jitter  performance  if  applied  to 
a  much  larger  network  with  a  higher  degree  of  connectivity. 

Actually  the  advantages  of  PRC  are  not  limited  to  the  reduction  of 
measurement  jitter.  Comparison  of  a  directed  control  system  with  Double  Endedness 
and  ICEM&C  (Figure  C11GP)  and  the  same  system  with  PRC  added  (Figure  C12GP*)  and 
then  comparing  this  latter  against  the  plots  for  the  same  two  systems  with 
self-organization  added  (Figures  C15GP  and  C16GP*)  indicates  that  the  greatest 
contribution  of  Phase  Reference  Combining  might  be  in  its  contribution  to 
automatic  self-organization. 

Finally  we  look  at  the  question  should  the  system  be  referenced  to 
precise  time.  All  present  commercial  digital  communication  systems  use  Directed 
Control  (Master  Slave).  They  all  use  redundant  atomic  clocks  as  their  ultimate 
master  frequency  source.  When  an  entire  network  is  locked  to  a  single  master  it 
appears  to  be  quite  economical  to  make  that  one  (or  a  few  when  alternate  masters 
are  to  be  selected  when  self-organization  is  provided)  master  clock  of  high 
quality.  If  low  quality  clocks  are  to  be  controlled  in  a  master  slave 
configuration  making  the  master  of  similar  low  quality  forces  the  control 
mechanism  throughout  the  network  to  be  engineered  to  handle  twice  the  total 
frequency  and  phase  variation  as  would  be  required  if  the  system  master  were  at 
least  an  order  of  magnitude  better  quality.  Thus  it  is  concluded  that  it  is  well 
worth  putting  a  very  high  quality  clock  in  the  position  of  ultimate  normal  master 
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clock  and  in  a  system  having  self-organization  capability  to  scatter  some 
additional  good  quality  clocks  throughout  the  network. 

Once  a  good  quality  clock  is  available  it  is  quite  simple  to  coordinate 
its  time  with  UTC(USNO).  Actually  the  requirements  of  Federal  Standard  1002 
dictates  that  this  be  the  case.  It  is  obvious  that  the  DCS  will  end  up 
interfacing  other  communication  networks  both  Military  and  Commercial  and  thus  the 
direction  of  Federal  Standard  1002  cannot  be  denied. 
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APENOIX  B 


LOOP  PARAMETER  SELECTION 


As  indicated  in  Paragraph  3. 1.1. 3,  the  problem  of  finding  the  "best" 
loop  parameters  for  use  in  the  nodal  clock  is  not  at  all  straightforward.  The 
difficulty  centers  around  the  fact  that  there  are  a  great  many  performance  indices 
and  all  of  them  should  be  examined  for  a  thorough  analysis.  In  this  appendix  we 
will  study  a  few  of  the  more  significant  performance  figures  as  a  function  of  the 
loop  parameters  £  and  a>n,  and  indicate  the  trade-offs  involved.  For  the  sake  of 
brevity,  we  will  restrict  the  analysis  to  the  tracking  mode  of  a  type  two  loop. 
Techniques  similar  to  those  employed  here  could  be  applied  to  the  type  one  loop 
and  the  acquisition  mode  of  the  type  two  loop,  furnishing  similar  results. 

The  performance  measures  which  will  be  examined  are: 

i)  Frequency  error  due  to  a  phase  step  input 

ii)  Phase  error  due  to  a  frequency  step  input 

iii)  Frequency  error  due  to  a  drifting  local  clock 

iv)  Phase  error  due  to  a  drifting  local  clock 

v)  Attenuation  of  daily  jitter. 

To  allow  the  results  developed  here  to  be  applied  to  a  network  of  any 
nominal  frequency,  phase  will  be  expressed  in  seconds  and  frequency  errors 
expressed  as  fractional  frequency  error.  That  is,  an  absolute  frequency  error  Af 
will  be  expressed  as  Af/f^  where  f^  is  the  network  frequency. 

The  general  model  for  a  type  two  loop  is  shown  below: 


It  is  easily  shown  that 


(s)  = 


Fq  I*) 


s‘  <?i  (S) 

+  2  {t*»n  s  ♦ 


♦  <onZ  s|  ^  (S) 

s2  ♦  2^  s  ♦«nZ 


and 


«o  (s) 


Fo 


2  to>n  s  +  wn 


*1  (s)  Fi  (s)  s2  ♦  2(o>  s  +  w  2 


(B-l ) 


(B-2) 


(B-3) 


where  o*n  =  ySTv 
and  i  -  >/Kv/4a 

Since  we  are  examining  only  the  tracking  performance  of  a  type  two 
loop,  we  will  always  require 

i  >1. 


when  this  condition  is  satisfied  we  find  that 
2 

S  ♦  2  Up  S  +  o»n2  =  (s  ♦  a)  (s  +  b) 


where 

a 

=  "n  U 

(B-4) 

and 

b 

=  o»n  (C  +  \/{2-l) 

(B-5) 

It  is 

easy  to 

see  that 

ab 

•“n2 

(B-6) 

The  above  relationships  will  be  very  useful  to  us  in  the  analysis  to 
follow  and  are  presented  here  to  avoid  needless  repetition. 

i)  Output  frequency  error  due  to  a  phase  step  input. 

Let 


(t)  =  0,  t<0 
=  t>0 


B-2 


Then  we  have  that 


0i  (s)  -  0/S 

Substituting  into  Equation  (B-2)  we  find  that 

(2  {to  s  +  w2)0 
Fo  (s)  =  ■  -{s'  ♦  aj"( s  ♦  b) 

(s  +  a)  (s  +  b)  (s  +  a)  (s  +  b) 
Finding  the  inverse  Laplace  transform  yields 


fo  <‘>  . 


b-a 


.  -bt  -at 
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-at  -bt 
e  -e 


It  can  be  shown  that  for  {>1,  the  second  term  is  negligible  in 
comparison  to  the  first  until  t  is  greater  than  2/w  From  this  equation  it  is 
easily  seen  that  the  peak  frequency  error  occurs  at  t  =  0  with  value 

^2  =  n«>no 

This  relationship  is  displayed  graphically  f or  0  =  1  ns  in  Figure  B-l. 
Clearly,  the  results  can  be  scaled  linearly  for  other  phase  offsets. 

ii)  Phase  error  due  to  a  fractional  frequency  step  input 
Let 

fi  (t)  ■  0,  t  <  0 

=  Af/fN,  t  >  0 

Then 


^  (s) 


Af/fN 


B-3 


0,  (s)  . 

s2 

Substituting  into  Equation  (B-l), 


Oc  (s) 


This  transforms  readily  to 


8c  (  t) 


Af/fN  (e-at-e-bt) 


We  can  find  that 


t  _  lna  -  Inb 
a-b 

So  the  peak  phase  error  is 


(e'atm  -e‘btm) 


where 

.  _  lna  -  Inb 

m  "  a-b 

Figure  B-2  shows  8^p  as  a  function  of  £  and  wn  for  Af/fN  = 
iO-9.  Again,  these  results  can  be  scaled  for  other  frequency  steps. 

iii)  Fractional  frequency  error  due  to  a  drifting  lock  clock. 

This  problem  can  be  solved  quite  readily  if  we  make  use  of  the 
fact  that  the  performance  of  the  loop  with  a  stable  reference 
and  a  drifting  local  clock  is  identical  to  that  obtained  with  a 


drifting  reference  and  a  stable  local  clock.  Since  we  are  only 
interested  in  the  error  between  the  input  and  output 
frequencies,  we  can  let  the  input  drift  and  then  examine  the 
amount  by  which  the  output  fails  to  follow  the  drift.  This 
difference  will  be  the  frequency  error.  Using  this  approach, 
we  have 


where  B  is  the  drift  rate  expressed  in  parts/second 


Substituting  into  Equation  (B-3),  we  find  that 


So  that,  after  some  manipulation 


where  k 


Clearly,  then,  the  error  between  the  input  and  output 
frequencies  (since  f.  (t)  =  Bt)  is 


with  k^  ^2 »  ar>d  tm  as  defined  above. 

This  peak  error  is  plotted  in  Figure  B-3  as  a  function  of  £  and 


Phase  error  due  to  a  drifting  local  clock. 
Using  the  same  approach  as  above,  let 
(t)  =  Bt 

so  that 

0,  (s)  =  B/s3 

Equation  (B-l)  then  becomes 

0E  (s)  =  - ^ - 

s  (s  +  a)  (s  +  b) 

Which,  in  turn,  implies  that 


0E  (t) 


b 

b-a 
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(B-10) 


It  is  easy  to  show  that  Equation  (B-10)  has  a  peak  value  equal 
to  its  steady-state  value.  Recalling  that  ab  =<Jn  2,  this 
is 


^Ess  =  B/a»n  (B-ll) 

Equation  (B-ll)  is  displayed  in  Figure  B-4  for  B  =  10"^/day 
(=  1.16  x  10-15/s). 


I 


v)  Attenuation  of  Daily  Jitter  in  the  Coasting  Mode 

When  the  type  two  loop  is  operating  without  a  reference,  its 
output  frequency  is 


F0  (s)  = 


“nV*) 

s2  *  2{%  s  *  ^ 


This  yields  a  transfer  function 
I  H  M|  = 


Fq 


/u4  +  (4£2  -2)fxj2<un2  +<un4 


Since  it  is  expected  that  daily  jitter  will  be  the  most 
troublesome,  we  are  most  concerned  with 

a)  r2 


|H  (<V|  V"D4  *  (4{2  -2) 


2  2 
'n 


u>„  +  O 


n 


where 


\-5 


=  7.2685  x  10  rad/sec 


it  | H  (wQ)  |  is  the  amplification  of  daily  jitter  provided 
by  the  loop,  then  the  attenuation  is  given  by 


A  plot  of  this  equation  is  shown  in  Figure  B-5  as  a  function 
ofwn  for  several  values  of  {  . 


SUMMARY 


Examining  Figures  B-l  through  B-5,  some  interesting  observations  can  be 
made.  The  most  important  of  these  is  that  improvements  in  one  performance  aspect 
will  result  in  degradation  of  another.  For  example,  in  Figure  B-l,  we  see  that 
decreasing^  and/or  £  decreases  the  frequency  error  due  to  a  phase  step,  while 
in  Figure  B-2  it  is  clear  that  decreasing  cjn  and/or  £  increases  the  phase  error 
due  to  a  frequency  step.  It  should  also  be  noted  that  some  performance  measures 
shoud  be  considered  more  carefully  than  others.  For  example,  peak  frequency  error 
due  to  a  drifting  clock  is  three  orders  of  magnitude  smaller  than  that  due  to  a 
1  /as  phase  step  in  the  range  of  interest.  Also,  steady-state  errors  should 
probably  be  weighted  more  heavily  than  peak  errors.  All  these  observations  lead 
to  the  conclusion  that  there  really  are  no  "optimum"  loop  parameters.  Rather,  we 
must  make  an  intelligent  selection  based  on  engineering  judgment  and  an  intuitive 
feel  for  the  trade-offs  involved. 


Figure  B-l.  Peak  Fractional  Frequency  Error  for  a  Type 
Loop  Due  to  a  1  /is  Phase  Step 


Attenuation  of  Daily  Jitter  (  fis  =  7.2685  x  10°  Rad/Sec) 
for  a  Type  Two  Loop  in  the  Coasting  Mode 
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11GF*  DC+DE+ICEM&C  Frequency  plot  for  directed  control  with  C-42 

double-ended  and  independence  of 
measurement  and  correction.  General 
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and  self-organizing.  General  stress 
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13GF  DC+SO  Frequency  plot  for  directed  control  with  C-46 

type  2  and  self-organizing.  General 
stress  scenario. 

14GP  OC+DE+SO  Phase  plot  for  directed  control  with  C-47 

double-ended,  and  self-organizing. 

General  stress  scenario. 

14GF  DC+DE+SO  Frequency  plot  for  directed  control  with  C-48 

double-ended,  and  self-organizing. 

General  stress  scenario. 
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DC+DE+ICEM&C+SO 

Phase  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  General  stress  scenario 
(with  jitter). 
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Frequency  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  General  stress  scenario 
(with  jitter). 

C-52 

16GP* 

DC+DE+ICEM8.C+ 
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Phase  plot  for  directed  control  with 
double-ended,  independence  of  measurement 
and  correction,  phase  reference  combining 
and  self-organizing.  General  stress 
scenario  (with  jitter). 
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Frequency  plot  for  directed  control  with 
double-ended,  independence  of  measurement 
and  correction,  phase  reference  combining 
and  self-organizing.  General  stress 
scenario  (with  jitter). 
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Phase  plot  for  directed  control  with  type  1 
loop  (mutual  sync  loop  parameters).  Low 
level  stress  scenario. 
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DC-1 

Frequency  plot  for  directed  control  with 
type  1  loop  (mutual  sync  loop  parameters). 
Low  level  stress  scenario. 
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Phase  plot  for  directed  control  with  type  2 
loop.  Low  level  stress  scenario. 
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Frequency  plot  for  directed  control  with 
type  2  loop.  Low  level  stress  scenario. 
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Phase  plot  for  mutual  control  with  equal 
weighting.  Low  level  stress  scenario. 
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Frequency  plot  for  mutual  control  with 
equal  weighting.  Low  level  stress 
scenario. 
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Phase  plot  for  mutual  control  with  unequal 
weighting.  Low  level  stress  scenario. 
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Frequency  plot  for  mutual  control  with 
unequal  weighting.  Low  level  stress 
scenario. 
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Phase  plot  for  mutual  control  with  a 
master  and  equal  weighting.  Low  level 
stress  scenario. 


MC+M+EW 


Frequency  plot  for  mutual  control  with  a 
master  and  equal  weighting.  Low  level 
stress  scenario. 


MC+M+UEW 


MC+M+UEW 


MC+EW+DOS 


MC+EW+DOS 


Phase  plot  for  mutual  control  with  a 
master  and  unequal  weighting.  Low  level 
stress  scenario. 

Frequency  plot  for  mutual  control  with  a 
master  and  unequal  weighting.  Low  level 
stress  scenario. 

Phase  plot  for  mutual  control  with  dropout 
smoothing  (and  equal  weighting.)  Low 
level  stress  scenario. 

Frequency  plot  for  mutual  control  with 
dropout  smoothing  (and  equal  weighting.) 
Low  level  stress  scenario. 


DC+DE 


Phase  plot  for  directed  control  with  type  2 
loop  and  double-ended.  Low  level  stress 
scenario. 


DC+OE 


MC+DE+EW 


Frequency  plot  for  directed  control  with 
type  2  loop  and  double-ended.  Low  level 
stress  scenario. 

Phase  plot  for  mutual  control  with  equal 
weighting  and  double-ended.  Low  level 
stress  scenario. 


MC+DE+EW 


Frequency  plot  for  mutual  control  with 
equal  weighting  and  double-ended.  Low 
level  stress  scenario. 


MC+M+DE+UEW+DOS 


MC+M+DE+UEW+DOS 


Phase  plot  for  mutual  control  with  a 
master,  unequal  weighting,  dropout 
smoothing,  and  double-ended.  Low  level 
stress  scenario. 

Frequency  plot  for  mutual  control  with  a 
master,  unequal  weighting,  dropout 
smoothing,  and  double-ended.  Low  level 
stress  scenario. 
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Phase  plot  for  directed  control  with  C-75 

double-ended  and  independence  of 
measurement  and  correction.  Low  level 
stress  scenario. 

Frequency  plot  for  directed  control  with  C-76 
double-ended  and  independence  of 
measurement  and  correction.  Low  level 
stress  scenario. 

Phase  plot  for  directed  control  with  C-77 

double-ended  independence  of  measurement 
and  correction.  Low  level  stress  scenario 
(with  jitter). 

Frequency  plot  for  directed  control  with  C-7S 
double-ended  independence  of  measurement 
and  correction.  Low  level  stress  scenario 
(with  jitter). 
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14LP 

14LF 


DODE+ICEM&C+PRC  Phase  plot  for  directed  control  with  C-79 

double-ended,  independence  of  measurement 
and  correction,  and  phase  reference 
combining.  Low  level  stress  scenario 
( with  jitter) . 

DC+DE+ICEM&C+PRC  Frequency  plot  for  directed  control  with  C-80 

double-ended,  independence  of  measurement 
and  correction,  and  phase  reference 
combining.  Low  level  stress  scenario 
(with  jitter). 


DC+SO 

DC+SO 

DC+DE+SO 


DC+DE+SO 


Phase  plot  for  directed  control  with  type  2  C-81 

and  self-organizing.  Low  level  stress 
scenario. 

Frequency  plot  for  directed  control  with  C-82 
type  2  and  self-organizing.  Low  level 
stress  scenario. 

Phase  plot  for  directed  control  with  C-83 

double-ended,  and  self-organizing.  Low 
level  stress  scenario. 

Frequency  plot  for  directed  control  with  C-84 
double-ended,  and  self-organizing.  Low 
level  stress  scenario. 
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Phase  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  Low  level  stress  scenario. 

Frequency  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  Low  level  stress  scenario. 
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DC+DE+ICEM&C+SO 

Phase  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  Low  level  stress  scenario 
( with  jitter) . 
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15LF* 
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DC+DE+ICEMtC+SO 

Frequency  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  Low  level  stress  scenario 
(with  jitter). 
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DC+DE+ICEM8.C+ 

PRC+SO 

Phase  plot  for  directed  control  with 
double-ended,  independence  of  measurement 
and  correction,  phase  reference  combining 
and  self-organizing.  Low  level  stress 
scenario  (with  jitter). 

C-89 

16LF* 

L 

r 

DC+DE+ICEM&C+ 

PRC+SO 

Frequency  plot  for  directed  control  with 
double-ended,  independence  of  measurement 
and  correction,  phase  reference  combining 
and  self-organizing.  Low  level  stress 
scenario  (with  jitter). 
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Phase  plot  for  directed  control  with  type  1 
loop  (mutual  sync  loop  parameters).  High 
level  stress  scenario. 

C-91 

1HF 
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DC-1 

Frequency  plot  for  directed  control  with 
type  1  loop  (mutual  sync  loop  parameters). 
High  level  stress  scenario. 

C-92 

-  2HP 

DC- 2 

Phase  plot  for  directed  control  with  type  2 
loop.  High  level  stress  scenario. 

C-93 

2HF 

\ 

DC- 2 

Frequency  plot  for  directed  control  with 
type  2  loop.  High  level  stress  scenario. 
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3HP 

MC+EW 

Phase  plot  for  mutual  control  with  equal 

C-95 

weighting.  High  level  stress  scenario. 
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3HF 

MC+EW 

Frequency  plot  for  mutual  control  with 
equal  weighting.  High  level  stress 
scenario. 

C-96 
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MC+UEW 

Phase  plot  for  mutual  control  with  unequal 
weighting.  High  level  stress  scenario. 

C-97 

4HF 

MC+UEW 

Frequency  plot  for  mutual  control  with 
unequal  weighting.  High  level  stress 
scenario. 
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5HP 

MC+M+EW 

Phase  plot  for  mutual  control  with  a 
master  and  equal  weighting.  High  level 
stress  scenario. 

C-99 

5HF 

MC+M+EW 

Frequency  plot  for  mutual  control  with  a 
master  and  equal  weighting.  High  level 
stress  scenario. 

C-100 

6HP 

MC+M+UEW 

Phase  plot  for  mutual  control  with  a 
ster  and  unequal  weighting.  High  level 
stress  scenario. 

C-101 

6HF 

MC+M+UEW 

Frequency  plot  for  mutual  control  with  a 
master  and  unequal  weighting.  High  level 
stress  scenario. 

C-102 

7HP 

MC+EW+DOS 

Phase  plot  for  mutual  control  with  dropout 
smoothing  (and  equal  weighting).  High 
level  stress  scenario. 

C-103 

7HF 

MC+EW+DOS 

Frequency  plot  for  mutual  control  with 
dropout  smoothing  (and  equal  weighting). 

High  level  stress  scenario. 

C-104 

8HP 

DC+DE 

Phase  plot  for  directed  control  with 
type  2  loop  and  double-ended.  High  level 
stress  scenario. 

C-105 

8HF 

DC+DE 

Frequency  plot  for  directed  control  with 
type  2  loop  and  double-ended.  High  level 
stress  scenario. 
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9HP 

MC+DE+EW 

Phase  plot  for  mutual  control  with  equal 
weighting  and  double-ended.  High  level 
stress  scenario. 

C-107 

9HF 

MC+DE+EW 

Frequency  plot  for  mutual  control  with 
equal  weighting  and  double-ended.  High 
level  stress  scenario. 
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DC+OE+ICEMSiC 


DC+DE+ICEM8.C 


Phase  plot  for  mutual  control  with  a 
master,  unequal  weighting,  dropout 
smoothing  and  double-ended.  High  level 
stress  scenario. 

Frequency  plot  for  mutual  control  with  a 
master,  unequal  weighting,  dropout 
smoothing  and  double-ended.  High  level 
stress  scenario. 

Phase  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction.  High  level 
stress  scenario. 

Frequency  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction.  High  level 
stress  scenario. 


DC+DE+ICEMtC 


DC+DE+ICEMiC 


DC+DE+ICEM8.C+PRC 


DC+DE+ICEM&C+PRC 


DC+SO 


Phase  plot  for  directed  control  with 
double-ended  independence  of  measurement 
and  correction.  High  level  stress 
scenario  (with  jitter). 

Frequency  plot  for  directed  control  with 
double-ended  independence  of  measurement 
and  correction.  High  level  stress 
scenario  (with  jitter). 

Phase  plot  for  directed  control  with 
double-ended,  independence  of  measurement 
and  correction,  and  phase  reference 
combining.  High  level  stress  scenario 
(with  jitter). 

Frequency  plot  for  directed  control  with 
double-ended,  independence  of  measurement 
and  correction,  and  phase  reference 
combining.  High  level  stress  scenario 
(with  jitter). 

Phase  plot  for  directed  control  with 
type  2  and  self-organizing.  High  level 
stress  scenario. 
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Frequency  plot  for  directed  control  with 
type  2  and  self-organizing.  High  level 
stress  scenario. 
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DC+DE+ICEM&C+SO 


DC+DE+ICEM&C+ 

PRC+SO 


DC+DE+ICEM&C+ 

PRC+SO 


Phase  plot  for  directed  control  with 
double-ended,  and  self-organizing.  High 
level  stress  scenario. 

Frequency  plot  for  directed  control  with 
double-ended,  and  self-organizing.  High 
level  stress  scenario. 

Phase  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  High  level  stress  scenario. 

Frequency  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  High  level  stress  scenario. 

Phase  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  High  level  stress  scenario 
(with  jitter). 

Frequency  plot  for  directed  control  with 
double-ended  and  independence  of 
measurement  and  correction  and  self¬ 
organizing.  High  level  stress  scenario 
(with  jitter). 

Phase  plot  for  directed  control  with 
double-ended,  independence  of  measurement 
and  correction,  phase  reference  combining 
and  self-organizing.  High  level  stress 
scenario  (with  jitter). 

Frequency  plot  for  directed  control  with 
double-ended,  independence  of  measurement 
and  correction,  phase  reference  combining 
and  self-organizing.  High  level  stress 
scenario  (with  jitter). 
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Appendix  C 

Phase  and  Frequency  Plots 


This  appendix  contains  all  the  plots  of  phase  and  frequency  variations 
produced  by  the  computer  simulations  which  are  described  in  Section  3.2  of  this 
report.  The  following  List  of  Illustrations  is  divided  into  three  groups.  The 
first  group  contains  the  results  of  phase  and  frequency  plots  resulting  from  the 
use  of  the  general  stress  scenerios  described  in  Section  3.2.3.10.  The  notation 
Figure  C-1GP  on  a  graph  stands  for  the  number  1  feature  combination  listed  in 
Table  C-I,  The  G  stands  for  General  Scenerio  and  the  P  for  Phase  Plots.  The  next 
plot  is  designated  IGF  for  Feature  Combination  1,  General  Scenerio,  Frequency 
Plots,  etc. 

The  second  group  is  result  of  the  application  of  the  Low  Level  Stress 

Scenerio.  Thus  14  LP  is  the  graph  of  results  of  Feature  Combination  14  (shown  in 

Table  C-I),  for  the  Low  Level  Stress  Scenerio  described  in  Section  3.2.3.10  and  is 
the  phase  plot. 

The  plots  were  made  on  a  plotter  which  produced  the  symbols  shown  in 
Table  C- II.  That  table  relates  the  plot  symbols  to  the  node  numbers  which  are 
contained  in  the  center  of  each  circle  in  Figure  C-l. 

The  third  group  uses  H  instead  of  G  or  L  to  denote  the  High  Level 
Stress  Scenerio  was  used. 

The  Figure  C-8GP'  and  8GF*  denote  that  the  drop-in  smoothing  and 
coasting  features  were  not  used  for  that  simulation  as  discussed  in  Section  2.2.8. 

The  figure  numbers  which  end  in  an  asterick  such  as  15  GP*,  etc.,  were 

run  with  jitter  added  to  the  stress  as  discussed  in  Section  3. 2. 3. 8. 


C-ll 


Table  C-I.  Feature  Combinations  for  Simulations 


This  is  a  repeat  of  Table  3.2.3.11.  It  provides  definition  of  the  relationship 
between  Run  Numbers  and  Feature  Combination  used  throughout  this  report. 


1. 

DC-1 

Directed  control  with  Type  1  loop  (mutual  sync  loop 
parameters) . 

2. 

DC-2 

Directed  control  with  Type  2  loop. 

3. 

MOtW 

Mutual  control  with  equal  weighting. 

4. 

MC+UEW 

Mutual  control  with  unequal  weighting. 

5. 

MC+M+EW 

Mutual  control  with  a  master  and  equal  weighting. 

6. 

MC  +  M+-UEW 

Mutual  control  with  a  master  and  unequal  weighting. 

7. 

MC+EW+DOS 

Mutual  control  with  dropout  smoothing  (and  equal 
wei ght i ng) . 

8. 

DC+DE 

Directed  control  with  Type  2  loop  and  double-ended. 

9. 

MODE+EW 

Mutual  control  with  equal  weighting  and  double-ended. 

10. 

MC+M+DE+  UEW+DOS 

Mutual  control  with  a  master,  unequal  weighting,  dropout 
smoothing,  and  double-ended. 

11. 

DC+DE+ I CEM&C 

Directed  control  with  double-ended  and  independence  of 
measurement  and  correction. 

12. 

DC+DE+ ICEM&C+PRC 

Directed  control  with  double-ended,  independence  of 
measurement  and  correction,  and  phase  reference 
combi ni ng . 

SELF -ORGAN I Z I NG  RUNS 

13. 

DC+SO 

Directed  control  with  Type  2. 

14. 

DC+DE+SO 

Directed  control  with  double-ended. 

15. 

DC+  DE+- 1  CEM&C+SO 

Repeat  Run  No.  11. 

16. 

DC+DE+ 1CEM&C+PRC+S0 

Repeat  Run  No.  12. 
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The  numbers  refer  to  the  Node  numbers  as  shown  in  the  network  topology  shown  in 
Figure  C-l  and  described  in  Section  3. 2. 3. 2. 


Phase  plot  for  directed  control  with  type  1  loop  (mutual  sync  loop 
parameters).  General  stress  scenario. 
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Frequency  plot  for  directed  control  with  type  1  loop  (mutual  sync 
loop  parameters).  General  stress  scenario. 


cy  plot  for  directed  control  with  type  2  loop.  General 
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Phase  plot  for  directed  control  with  type  2  loop  and 
without  drop-in  smoothing  and  coasting.  General  stress 
scenario. 
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Phase  plot  for  mutual  control  with  equal  weighting.  General  stress 
scenario. 
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5GF  Frequency  plot  for  mutual  control 
General  stress  scenario. 


Fig.  C  6GP  Phase  plot  for  mutual  control  with  a  master  and  unequal  weighting 
MC+CHUEW  General  stress  scenario. 
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GP  Phase  plot  for  mutual  control  with  dropout  smoothing 
weighting. )  General  stress  scenario. 


Fig.  C  7GF  Frequency  plot  for  mutual  control  with  dropout  smoothing 
MC+EW+DOS  equal  weighting. )  General  stress  scenario. 


Fig.  C  8GP1  Phase  plot  for  directed  control  with  type  2  loop,  double-ended 

and  without  drop-in  smoothing  and  coasting.  General  stress 
DC+DE  scenario. 


Frequency  plot  for  directed  control  with  type  2 
ended  and  without  drop-in  smoothing  and  coast 


Fig.  C  9GP  Phase  plot  for  mutual  control  with  equal  weighting  and 
double-ended.  General  stress  scenario. 

MC+DE+EW 


Fig.  C  11GP  Phase  plot  for  directed  control  with  double-ended  and  independence 
measurement  and  correction.  General  stress  scenario. 


Phase 


Fig.  C  12GP-  Phase  plot  for  directed  control  with  double-ended,  independence 

of  measurement  and  correction,  and  phase  reference  combining. 
DC+DE+ICEM4C+PRC  General  stress  scenario  (with  jitter). 
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Phase  plot  for  directed  control  with  type  2  and  self 
organizing.  General  stress  scenario. 


F requency  plot  for  directed  control  wi 
organizing.  General  stress  scenario 


14GF  Frequency  plot  for  directed  control  with  double-ended,  and  self- 
organizipg.  General  stress  scenario. 


Fig.  C  15GF  Frequency  plot  for  directed  control  with  double-ended  and  independence 
of  measurement  and  correction  and  self  organizing.  General  stress 
DC+DE+ICEM&C+SO  scenario. 
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cy  plot  for  directed  control  with  double-ended  and  independence 
urement  and  correction  ar.d  self  organizing.  General 
cenario  (with  jitter). 


Fig.  C  1  LP  Phase  plot  for  directed  control  with  type  1  loop  (mutual  sync  loop 
parameters).  Low  level  stress  scenario. 


Frequency  plot  for  directed  control  with  type  1  loop  (mutual  sync 
loop  parameters).  Low  level  stress  scenario. 


plot  for  mutual  control  with  equal  weighting.  Low  level 
scenario. 


Frequency  plot  for  mutual  control  with  unequal  weighting.  Low 
level  stress  scenario. 


5LP  Phase  plot  for  mutual  control  with  a  maste 
Low  level  stress  scenario. 
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Fig.  C  7LP  Phase  plot  for  mutual  control  with  dropout  smoothing 
Mr+ru+nnc  weighting.  )  Low  level  stress  scenario. 


directed  control  with  type  2  loop  and  double 
rel  stress  scenario. 


ible-ended.  Low  level  stres 


Fig.  C  9LF  Frequency  plot  for  mutual  control  with  equal  weighting  and 
MC+DE+EW  double-ended.  Low  level  stress  scenario. 


Fig.  C  11LF*  Frequency  plot  for  directed  control  with  double-ended  and 

independence  of  measurement  and  correction.  Low  level  stress 
DC+DE+ICEM&C  scenario  (with  jitter). 


Fig.  C  12LP*  Phase  plot  for  directed  control  with  double-ended,  independence 

of  measurement  and  correction,  and  phase  reference  combining. 
DC+DE+ICEMSC+PRC  Low  level  stress  scenario  (with  jitter). 
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Phase  plot  for  directed  control  with  type  2  and  self¬ 
organizing.  Low  level  stress  scenario. 
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14LP  Phase  plot  for  directed  control  with  double-ended,  and  self 
organizing.  Low  level  stress  scenario. 


Fig.  C  15LP  Phase  plot  for  directed  control  with  double-ended  and  independence 

of  measurement  and  correction  and  self  organizing.  Low  level  stress 

DC+DE+ICEM4C+S0  scenario. 


DC+DE+ICEM&C+SO 


Fig.  C  15LF*  Frequency  plot  for  directed  control  with  double-ended  and 

independence  of  measurement  and  correction  and  self 
DC+DE+ICEM&OSO  organizing.  Low  level  stress  scenario  (with  jitter). 


Fig.  C  16LP*  Phase  plot  for  directed  control  with  double-ended,  independence  of 

measurement  and  correction,  phase  reference  combining  and  self 
DC+DE+ICEM&C+PRC+SO  organizing.  Low  level  stress  scenario  (with  jitter). 
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F requency  plot  for  directed  control 
sync  loop  parameters).  High  level 


plot  for  directed  control  with  type  2  loop.  High  level 
scenario. 
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Frequency  plot  for  directed  control  with  type  2  loop.  High  level 
stress  scenario. 


Frequency  plot  for  mutual  control  with  equal  weighting.  High 
level  stress  scenario. 
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Frequency  plot  for  mutual  control  with  unequal 
level  stress  scenario. 
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mutual  control  with  a  master  and  equal  weighting 
ss  scenario. 


Fig.  C  5HF  Frequency  plot  for  mutual  control  with  a  master  and  equal  weighting 
MT4.M4.ru  High  level  stress  scenario. 


mutual  control  with  a  master  and  unequal  weighting 
ss  scenario. 
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Fig.  C  6HF  Frequency  plot  for  mutual  control  with  a  master  and  unequal 
itfM.M4.tiru  weighting.  High  level  stress  scenario. 


Fig.  C  8HP  Phase  plot  for  directed  control  with  type  2  loop  and  double - 
Q0+0E  ended.  High  level  stress  scenario. 
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Fig.  C  11HF*  Frequency  plot  for  directed  control  with  double-ended  anc 

DC+DE+ICEM&C  independence  of  measurement  and  correction.  High  level 

stress  scenario  (with  jitter). 
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Fig.  C  13HP  Phase  plot  for  directed  control  with  type  2  and  self-organizing. 
DC+SO  High  level  stress  scenario. 
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Fig.  C  15HP*  Phase  plot  fo 

independence 
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Fig.  C  15HF*  Frequency  plot  for  directed  control  with  double-ended  and 

independence  of  measurement  and  correction  and  self 
DC+nF*ICEM,r-LS0  o:  izir  ’’  hie'  ’ 'str'  see _ ;.o  (•  jit' 
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APPENDIX  D 

DEFINITION  OF  HARMFUL  TRANSIENT 


APPENDIX  D 


DEFINITION  OF  HARMFUL  TRANSIENT 

We  are  concerned  with  transients  in  nodal  frequency.  In  the  strictest 
sense  the  function  of  the  nodal  synchronizer  is  to  provide  the  nodal  clocks  so 
that  the  digital  switching  and  multiplexing  operations  can  be  carried  out  without 
any  bit  slips  or  other  degradation  in  system  performance. 

If  a  frequency  transient  occurs  with  too  large  a  magnitude  and  too 
rapidly,  then  the  clock  recovery  loops  in  multiplexers  or  modems  that  must  use  the 
clock  can  suffer  bit  slips  or  a  signal-to-noise  ratio  degradation.  Whether  or  not 
a  given  transient  is  harmful  depends  on  the  bandwidth  of  the  clock  recovery  loop 
and  its  ability  to  track  these  variations  in  frequency.  Probably  the  principal 
problem  that  may  occur  is  that  certain  of  the  existing  equipments  that  will 
continue  to  be  used  on  the  digital  DCS  may  have  clock  recovery  loop  parameters 
that  would  be  susceptible  to  transients.  We  don't  know  the  exact  values  of  these 
parameters,  so  we  can't  accurately  differentiate  between  the  harmful  and 
nonharmful  transients.  However,  the  effect  of  choice  of  loop  parameters  will  be 
discussed  in  some  detail.  In  terms  of  equipment  to  be  procured  in  the  future,  one 
should  be  able  to  design  the  loops  such  that  no  problem  results  from  the  magnitude 
of  the  transients  expected. 

Even  though  it  is  very  difficult  to  distinguish  with  high  accuracy  the 
harmful  from  the  nonharmful  transients  we  shall  attempt  to  do  so.  A  transient  is 
defined  to  be  harmful  if  there  is  a  high  probability  of  it  causing  a  bit  slip  at 
some  nodes.  A  transient  is  defined  to  be  potentially  harmful  if  it  increases  the 

probability  of  a  bit  slip,  enough  such  that  a  combination  of  several  such 

/ 

transients  could  result  in  a  high  probability  of  a  bit  slip. 

Frequency  transients  will  have  the  most  severe  effects  on  the  clock 
recovery  loops  in  modems.  The  more  narrow  loop  bandwidths  will  have  more 


difficulty  tracking  frequency  transients.  The  largest  frequency  step  which  a 
second  order  loop  can  track  without  a  cycle  slip  is  equal  to  the  lock-in  range 
Ao>l_  where 

AwL  =  2  £*>„  (1) 

The  principal  reason  why  a  clock  recovery  loop  might  be  made  narrow  enough  to 
cause  a  problem  is  to  make  the  loop  capable  of  "coasting"  for  a  long  period  of 
time  without  a  reference  and  avoid  bit  slips.  The  time  constant  associated  with 
the  coast  mode  is 


T  = 


1 


(£  -  (C2-  1)1/2 


(2) 


for  £  >  1.  This  equation  can  also  be  written  as 


to  t  = 


J  ^-^«2-n1/2  • 


(3) 


CJ  w 

0  0 


where  is  the  clock  rate.  Equations  (1)  and  (3)  can  be  used  to  plot  curves  of 
wqt  versus  ^j=-  as  a  function  of  £.  These  are  shown  in  Figures  D-l  and  D-2. 


This  indicates,  of  course,  that  the  acceptable  fractional  frequency  step  Au^/cj^ 
is  a  function  of  (oQr(the  product  of  the  loop  time  constant  and  the  clock  rate) 
and  £.  Note  also  that  for  any  desired  value  of  <uQrthe  acceptable  magnitude  of 
the  frequency  transient  can  be  increased  by  increasing  £  (the  effect  of 
increasing  £  while  holding  r  constant  is  to  shorten  the  "short"  time  constant  of 
the  second  order  loop).  This  indicates  that  under  certain  conditions  there  exists 


a  value  of  £  which  allows  the  values  rand  ac^’evec** 


To  compute  this  value  of  ,  let 


P  =  (4) 

and 

a  =  wo  r  (5) 

Then  £  t  1  implies  that  apt  2.  By  combining  Equations  (3)  -  (5)  we  find  that  for 
any  desired  value  of  a  and  /3  where  ap  >  2 

{'M-j)2r2  (6) 

The  required  value  of  £  as  a  function  of  a/3  is  shown  in  Figure  D-3.  As  expected 
as  aP  increases  the  minimum  acceptable  value  of  £  in  order  to  avoid  bit  slip 
increases. 

To  get  an  indication  of  the  order  of  magnitude  of  these  parameters  we 
note  that  there  exist  tropo  modems  that  have  data  rates  of  several  Mb/s  and  a  long 
time  constant  of  several  seconds.  These  parameters  produce  a  cuqt product  of 
about  10®.  Figure  D-l  indicates  that  if  £  =  1  any  step  in  frequency  greater 
than  2  x  10"®  would  cause  a  slip.  However,  larger  frequency  steps  can  be 
accommodated  for  larger  values  of  £.  This  can  be  seen  from  Figure  D-3  where  we 
see  that  £'s  of  1.6,  5,  and  16  allow  accommodation  of  frequency  steps  of  10“7, 
10"®,  and  10"®,  respectively.  Though  we  do  not  know  the  loop  parameters  of 
all  existing  tropo  modems  that  may  be  used,  these  results  indicate  that  potential 
problems  may  be  caused  by  frequency  transients  as  large  as  2  x  10“®  and  that 
problems  probably  will  be  caused  by  transients  larger  than  10"7.  There 
certainly  exist  modems  in  the  current  inventory  (for  example  some  SATCOM  PSK 
modems)  which  have  much  larger  loop  bandwidths  than  these.  In  this  case  frequency 
transients  of  this  order  of  magnitude  would  cause  no  problem.  There  may  also 
exist  in  inventory,  modems  with  larger  (oq  rproducts,  though  we  do  not  know  of 


specific  examples.  Of  course,  in  this  case  the  requirements  would  be  more  severe 
than  we  have  postulated. 

For  equipment  that  has  not  yet  been  procured  one  should  be  able  to 

avoid  undesirable  effects  from  transients  provided  the  loop  parameters  are  chosen 

in  accordance  with  Figures  0-1  through  D-3.  It  is  likely  that  both  data  rates  and 

coasting  times  in  future  procurements  may  be  considerably  higher  than  those  of  the 

example  in  the  previous  paragraph.  Thus,  the  <o  t  products  may  be  several 

8  •  ° 

orders  of  magnitude  higher  than  10  .  For  a  given  size  transient  that  is 
expected,  this  simply  has  the  effect  of  increasing  the  minimum  value  of  £  that  can 
be  used.  Needless  to  say,  any  decrease  in  the  magnitude  of  such  transients  will 
reduce  this  minimum  £  thereby  increasing  the  flexibility  in  choosing  parameters 
for  modem  clock  recovery  loops. 

We  now  make  the  following  observations  (trying  to  be  somewhat 
conservative)  about  the  deleterious  effects  of  nodal  frequency  steps  of  various 
sizes: 


\(jJ 

(O 

0 


> 


<  10“7 


< 


<  10“9 


Definitely  harmful 

Potentially  harmful  and  it  causes  a 
signal-to-noise  ratio  degradation 

Unlikely  to  cause  observable  problems 


<  10  ^  -  No  effects  on  slip  rate  or  SNR 

The  numbers  given  above  are  "best  guesses"  and  do  not  have  a  high 
degree  of  accuracy.  We  are  limited  in  our  knowledge  of  the  loop  parameters  of  all 
current  equipment  that  will  be  used  on  the  digital  DCS.  What  DCA  needs  to  do  to 


improve  these  estimates  is  to  collect  this  data  for  all  equipment  that  will  be 
used  and  determine  from  Figures  D-l  and  D-2  what  the  maximum  acceptable  transient 
will  be.  In  addition,  in  any  specifications  for  new  equipment  the  clock  recovery 
loops  should  be  specified  to  handle  frequency  transients  of  a  certain  magnitude 
without  experiencing  bit  slips. 


